acceptodds
Under review as a conference paper at ICLR 2027

ReFrame: Elastic Expert–KV Memory Sharing for Mixture-of-Experts Serving

Abstract

In large mixture-of-experts serving systems with limited GPU memory, expert weights and key–value (KV) state compete for capacity, yet most existing systems statically partition this memory between them. Since only a subset of resident experts is activated per token, substantial expert memory can remain underutilized. Therefore, static GPU memory partitioning prevents the underutilized memory from being used by the KV cache, causing requests to wait for KV capacity and reducing overall memory efficiency. To address this limitation, we present ReFrame, a serving runtime that dynamically shares GPU memory between expert weights and KV cache while preserving sufficient expert residency. First, typed frame rebinding enables GPU memory to alternate between expert and KV cache while retaining the native tensor and block views required by each consumer. Second, a bounded allocation policy coordinates when and how much memory to borrow, confines each loan to the triggering request, and uses private capacity first to leave shared blocks available for other requests. Finally, we evaluate ReFrame on two mixed workloads with Qwen3-235B-A22B. Against CPU-offloading baselines, the complete system achieves geometric mean speedups of 2.75× over vLLM 0.29 and 2.94× over SGLang 0.5.17 in trace completion time. An independent matched-batch comparison with vLLM 0.29 yields 2.74×, and a separate comparison with its 2,048-token batch budget yields 3.15×.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Lanuage Model Inference, Mixture of Experts, Memory Management

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2QncOpenReview ↗
Rating
2
Confidence
1
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents ReFrame, an MoE serving runtime that dynamically shares GPU memory between expert weights and KV cache through typed frame rebinding and a bounded, request-isolated borrowing policy. Implemented on vLLM and evaluated on Qwen3-235B-A22B, it reports 2.75× and 2.94× completion-time speedups over vLLM and SGLang with CPU offloading.

Critical strengths

A typed frame rebinding mechanism that allows GPU memory to switch between expert weights and KV cache while keeping native tensor layouts and KV block tables unchanged, supported by an ownership protocol that prevents frames in use from being reclaimed.

A bounded, request-isolated borrowing policy that borrows only the smallest feasible number of frames, reserves them for the triggering request, and uses private blocks first to preserve shared capacity.

A full implementation on vLLM with a careful experimental protocol, including fresh processes for each run, reversed execution order, and frozen inputs, evaluated on a 235B-parameter MoE model.

Critical weaknesses

This is a pure systems paper with no learning or representation contribution, and it would be a better fit for venues such as OSDI, SOSP, EuroSys or MLSys.

The novelty is limited, since eLLM already supports elastic memory sharing between runtime tensors and KV cache, and Oneiros already remaps parameter memory to KV cache. The paper should clarify what is technically new beyond applying this idea to MoE expert weights.

All baselines use generic layer-wise CPU offloading, which transfers weights to the GPU on every forward pass, while ReFrame uses an MoE-aware expert cache. The paper should compare against MoE-specific offloading systems.

Questions for authors

See the above weaknesses.

3 comments

We thank the reviewer for highlighting scope, novelty and baseline coverage, and we have completed the revisions addressing the first and second issues.

W1: Venue Fit and Relevance to ICLR

ReFrame studies inference for a 235B MoE foundation model whose BF16 weights exceed four A100 GPUs' aggregate memory. Its ownership and allocation rules address competition between reconstructible expert weights and growing request-specific KV state. This contribution to foundation-model inference fits the selected primary area and aligns with the ICLR call.

W2: Novelty Beyond eLLM and Oneiros

We clarify novelty beyond eLLM and Oneiros in three parts: allocation semantics for expert/KV reuse, choice of experimental baseline, and the added Oneiros comparison.

(1) Allocation semantics for expert/KV reuse. A free native KV block is eligible only when every backing frame is ready and authorized for its request, including blocks that cross frame boundaries. Conversion first quiesces the outgoing owner and retires its publication. It then materializes the new role and publishes a new generation. Active KV stays protected. Request-scoped loans reserve temporary capacity for their owner. At a fixed state with feasible demand and independently selectable blocks, private-first minimizes shared-block consumption. The controller selects the smallest feasible prefix of a fixed candidate order, capped at two frames, using complete native blocks. These rules connect physical reuse to native tensor views and allocation decisions. The implementation uses explicit stream events assuming eager execution, serialized rank entry and complete producer/consumer stream coverage.

(2) Choice of experimental baseline. eLLM shares physical memory between activation buffers and KV tensors and supports KV offloading to CPU memory. Oneiros directly remaps parameter backing into KV capacity and thus more closely matches ReFrame's expert-to-KV reuse. Evaluating this mechanism with eLLM would require extending its published design to reclaim expert parameter memory. The additional measurements therefore focus on Oneiros. eLLM remains relevant to the broader discussion of elastic memory management.

(3) Added Oneiros comparison. The adaptation enables Oneiros's parameter/KV remapping and restoration paths. It is based on Oneiros's vLLM 0.7.3 fork. Six distinct runs completed 36 requests each, with 128 output tokens per request, for 216 requests and 27,648 tokens in total. The six four-GPU runs used 97.87 GPU-hours, including startup and shutdown but excluding failed attempts and development diagnostics.

All latencies are in seconds, ordered Oneiros / ReFrame. GPU memory utilization: 0.98/0.98. The last column gives the completion-time ratio.

WorkloadCompleted requestsCompletion time (s)TTFT p95 (s)TPOT p95 (s)Completion ratio
Mixed 136/3613,156.71 / 1,281.429,933.88 / 578.7521.50 / 2.9310.27×
Mixed 236/3612,118.22 / 1,247.789,126.46 / 434.1220.25 / 3.199.71×
Short requests36/3611,331.21 / 1,299.658,703.89 / 483.9918.30 / 3.448.72×
SummScreenFD + WildChat36/3611,957.31 / 1,102.069,044.55 / 412.7619.98 / 2.7510.85×
Arrival density 0.5×36/3612,084.81 / 1,749.548,349.80 / 483.4319.64 / 2.406.91×
Arrival density 1×36/3613,156.71 / 1,303.889,933.88 / 593.2021.50 / 3.1110.09×
Arrival density 2×36/3611,866.63 / 1,188.009,285.99 / 711.7719.94 / 3.219.99×

Density 1× reuses the Mixed 1 Oneiros run with a separate historical ReFrame reference.

From unrounded data, mixed-workload geometric-mean Oneiros/ReFrame ratios are 9.99× for completion, 19.00× for TTFT p95 and 6.83× for the content-span proxy p95.

Oneiros's published evaluation targets multi-tenant GH200 serving. Our experiments use an oversized BF16 MoE on four 80 GB A100s, with different transfer/computation costs.

W2: Part (1) of the response restates the mechanisms already described in the paper (ownership transitions, request-scoped loans, private-first allocation, and a prefix capped at two frames), but it does not identify what ReFrame can do that Oneiros cannot.

W3: The response does not address this point. The comparison against MoE-specific offloading systems such as MoE-Infinity, Fiddler, ProMoE or MoE-Lightning is still missing. This concern remains unresolved.

Thanks for your valuable suggestion, and we have completed the revisions addressing the second and third questions.

W2: Capability beyond Oneiros

ReFrame lets individual requests borrow expert frames for KV capacity while the MoE model continues serving. Oneiros already supports reversible parameter/KV remapping, including for active models. Its controller selects models and layers to remap, and its transfer engine uses the recurring order of layer execution. ReFrame extends this form of memory sharing with expert-level reallocation and request-specific allocation rules.

(1) Reclaim part of an active MoE layer. In our Qwen3-235B layout, one 36 MiB frame holds one expert's gate/up and down weights. When a request lacks KV blocks, ReFrame can lend eligible expert frames without remapping the remaining experts in those layers. The rank-wide expert cache and host-backed loading path continue to supply routed experts as residency changes. ReFrame can therefore adjust residency within a layer, whereas Oneiros's published policy reallocates capacity at layer granularity.

(2) Reserve reclaimed capacity for the request that needs it. Consider requests A and B sharing the same KV pool. If A triggers borrowing, B cannot allocate KV blocks backed by A's borrowed frames. A can use a block only when every backing frame is ready and authorized, even when the block spans multiple frames. After A releases the last live KV references, those frames become eligible for expert reconstruction without waiting for B to finish. The loan's lifetime is thus tied to A's KV use. Oneiros's published model/layer controller does not provide this request-owned expert-frame allocation rule.

The two-frame cap bounds the evaluated policy. ReFrame's added capability is to connect expert-level memory reallocation to a particular request's admission and KV lifetime. This gives the runtime control over which expert capacity is reassigned, which request can use it, and when it can return to expert residency.

W3: Comparison with MoE-Specific Offloading Systems

We detail the adaptation, measurement protocol, results, and scope caveats for the MoE-Infinity comparison.

MoE-Infinity comparison. The adaptation enables MoE-Infinity to serve Qwen3-235B with expert CPU offloading. It includes changes to expert eviction, scheduling, stream initialization, and last-position logit projection. Six distinct runs completed 36 requests each, with 128 output tokens per request, for 216 requests and 27,648 tokens in total. The six four-GPU runs used 132.53 GPU-hours, including startup and shutdown but excluding failed attempts and development diagnostics.

All latencies are in seconds, ordered MoE-Infinity / ReFrame. GPU memory utilization: 0.98/0.98. The last column gives the completion-time ratio.

WorkloadMoE-Infinity requestsCompletion time (s)TTFT p95 (s)TPOT p95 (s)Completion ratio
Mixed 136/3619,223.58 / 1,281.426,057.97 / 578.7596.83 / 2.9315.00×
Mixed 236/3622,188.08 / 1,247.787,494.04 / 434.12108.36 / 3.1917.78×
Short requests36/3612,814.18 / 1,299.652,433.89 / 483.9982.81 / 3.449.86×
SummScreenFD + WildChat36/3620,374.85 / 1,102.066,740.52 / 412.76100.66 / 2.7518.49×
Arrival density 0.5×36/3620,584.89 / 1,749.546,176.58 / 483.43104.19 / 2.4011.77×
Arrival density 1׆36/3619,223.58 / 1,303.886,057.97 / 593.2096.83 / 3.1114.74×
Arrival density 2×36/3621,319.75 / 1,188.006,495.36 / 711.77110.98 / 3.2117.95×

† Density 1× reuses the Mixed 1 MoE-Infinity run with a separate historical ReFrame reference.

From unrounded data, mixed-workload geometric-mean MoE-Infinity/ReFrame ratios are 16.33× for completion, 13.44× for TTFT p95 and 33.51× for TPOT p95.

MoE-Infinity uses activation-aware expert caching and prefetching to reduce offloading costs. ReFrame coordinates expert residency and KV admission under a shared memory budget. For Qwen3-235B in BF16, overlapping requests place both expert supply and KV capacity under pressure on four 80 GB GPUs. ReFrame’s joint management is intended to address this competition and may help explain the observed difference.

We are glad to have clarified ReFrame's capabilities beyond Oneiros and strengthened the comparison with MoE-specific offloading systems through the added MoE-Infinity experiments. We hope our response is satisfactory.

Reviewer_Q1m4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ReFrame is a runtime for memory-constrained MoE serving that temporarily reallocates GPU memory between expert weights and the KV cache. Typed frame rebinding preserves native expert-tensor and KV-block views during ownership changes. A bounded allocation policy reserves borrowed capacity for the triggering request and consumes private blocks first. On Qwen3-235B-A22B-Instruct-2507 with four A100 80-GB GPUs, the complete system achieves geometric-mean trace-completion speedups of 2.75× over vLLM 0.29 and 2.94× over SGLang 0.5.17 across two mixed workloads.

Critical strengths
  1. A coherent solution to a practical memory-management problem. Expert residency and KV capacity compete for limited GPU memory when model weights exceed device capacity. ReFrame exploits the distinction between reconstructible expert weights and live KV state, while preserving native execution interfaces. Its ownership protocol, request isolation, and private-first allocation provide a useful way to make these resources jointly manageable. I consider this integration valuable for deploying large MoE models on constrained hardware.

  2. Substantial complete-system benefits, with direct evidence that borrowing alleviates pressure. The framework-level advantage persists with a strengthened vLLM batch budget, an independent matched-batch comparison, a different long-document source, and several arrival densities.

Critical weaknesses
  1. The typical latency contribution of the central rebinding mechanism remains unclear. Figure 8 shows pairs retrospectively selected for the best long-request TTFT ratio. The selected same-configuration pair improves this metric by 12.17% and completion time by 5.55%, but the complete latency results for all four same configuration pairs are not shown. The separate Static ReFrame comparison reports a 1.24% aggregate improvement in long-request TTFT and a 2.29% regression in TPOT p95, although that control also differs in configuration. The large framework-level gains therefore do not establish the isolated benefit of rebinding. Full repeated-ablation results and a small fixed-partition comparison are important for judging whether dynamic sharing consistently improves upon the underlying expert-supply path.

  2. The scope of the incremental contribution is not sufficiently established. Prior parameter-remapping and memory-virtualization work motivates a clearer explanation of what ReFrame’s ownership and request-isolation semantics add, particularly relative to Oneiros. The evaluation also concentrates on one model, finite traces, nearly uniform long inputs, and fixed 128-token outputs. With one activation per busy episode, at most two candidate frames, and no new borrowing at decode edges, it is unclear how broadly the demonstrated policy applies. These limitations temper the significance of the contribution, even though the measured system-level gains are useful. A precise account of the new capabilities and policy boundaries would most strengthen my assessment; I do not require experiments against work available only as a preprint.

Questions for authors
  1. Clarification and validation of the reported component benefit: Could you provide TTFT, TPOT, and completion time for all four same-configuration rebinding-on/off pairs, including regressions? How does dynamic sharing compare with a fixed partition allocating one or two additional frames to KV from startup?

  2. Clarification of the technical contribution and implementation: What specific capabilities do typed frame rebinding and request-isolated allocation add beyond prior parameter remapping, especially Oneiros? Please explain how the ownership proof’s protection and readiness assumptions are enforced across asynchronous CUDA operations and ranks, and whether numerical-equivalence checks were performed.

  3. Clarification of the evaluated operating boundaries: How is a busy episode reset under continuous arrivals, and what happens when a loan owner needs more KV capacity during a long decode? What supports the 1,000ms threshold and two-frame cap? An explanation of the current implementation and any existing sensitivity evidence would help assess the submitted claims; a broad new serving benchmark is not necessary for this response.

4 comments

Thanks for your valuable suggestion, and we have completed the revisions addressing the second and third questions.

W2 / Q2: Clarification of the technical contribution and implementation

We organize the response into four parts: native-block eligibility, CUDA and rank coordination, numerical checks, and the Oneiros comparison.

(1) Native-block eligibility. A free block requires backing that is ready, generation-qualified and authorized for the request. Request-scoped loans reserve capacity for their owner. At a fixed state with feasible demand and independently selectable blocks, Proposition 2 shows that private-first minimizes shared-block use.

(2) CUDA and rank coordination. Each rank waits on explicit CUDA stream events and materialization before publishing a new owner-generation. Common logical block assignments use rank-local backing. Proposition 1 establishes frame-local protection under eager execution, serialized rank entry and complete producer/consumer stream coverage.

(3) Numerical checks. Against a TP8 reference, ReFrame matched 24/24 greedy tokens across three eight-token requests. Oneiros matched 16/17 comparable positions, with seven excluded after prefix divergence. Both systems exceeded the frozen full-vector error limits. These are cross-layout diagnostics.

(4) Operating boundaries (Q3). An episode resets only when both active and pending sets are empty. Continuous arrivals retain its one-activation allowance until then. Loan return does not reset it. Native scheduling continues.

All latencies are in seconds, ordered Oneiros / ReFrame. GPU memory utilization: 0.98/0.98. The last column gives the completion-time ratio.

WorkloadCompleted requestsCompletion time (s)TTFT p95 (s)TPOT p95 (s)Completion ratio
Mixed 136/3613,156.71 / 1,281.429,933.88 / 578.7521.50 / 2.9310.27×
Mixed 236/3612,118.22 / 1,247.789,126.46 / 434.1220.25 / 3.199.71×
Short requests36/3611,331.21 / 1,299.658,703.89 / 483.9918.30 / 3.448.72×
SummScreenFD + WildChat36/3611,957.31 / 1,102.069,044.55 / 412.7619.98 / 2.7510.85×
Arrival density 0.5×36/3612,084.81 / 1,749.548,349.80 / 483.4319.64 / 2.406.91×
Arrival density 1׆36/3613,156.71 / 1,303.889,933.88 / 593.2021.50 / 3.1110.09×
Arrival density 2×36/3611,866.63 / 1,188.009,285.99 / 711.7719.94 / 3.219.99×

† Density 1× reuses the Mixed 1 Oneiros run with a separate historical ReFrame reference.

From unrounded data, mixed-workload geometric-mean Oneiros/ReFrame ratios are 9.99× for completion, 19.00× for TTFT p95 and 6.83× for the TPOT p95.

Oneiros's published evaluation targets multi-tenant GH200 serving. Our experiments use an oversized BF16 MoE on four 80 GB A100s, with different transfer/computation costs.

W2 / Q3: Clarification of the evaluated operating boundaries

We clarify four aspects: episode reset, loan capacity during long decoding, the 1,000 ms threshold, and the two-frame cap.

An episode resets only when both active and pending sets are empty. Continuous arrivals retain its one-activation allowance until then. Loan return does not reset it. Native scheduling continues

During long decoding, the owner can use eligible free blocks in its loan and the shared pool. The loan remains assigned while its owner retains live KV references. Decode-edge borrowing is disabled. When capacity is insufficient, native scheduling applies, including possible preemption. The 1,000 ms threshold governs activation, not loan expiration.

Borrowing requires native allocation failure, 1,000 ms of allocator-visible pressure and at most six active decoders. This is pressure duration, not arrival-to-admission latency. Earlier development compared 1/2/4 s with borrowing-off at the same frame geometry. Two-pair geometric-mean TTFT p95/TPOT p95 ratios were 0.9594/1.0533, 1.0345/1.1073 and 1.0064/1.0277. The study stopped after 8/16 planned runs. Each setting exceeded the per-pair TPOT p95 limit of 1.05 at least once. Eight further pairs at 1 s on the same trace gave a TTFT p95 ratio of 0.9792 (one-sided 95% upper bound 1.0193). These results are exploratory. The submitted profile fixes 1,000 ms.

The cap is two 36 MiB frames (72 MiB per rank) per episode. Eq. (2) selects the smallest feasible prefix of the frozen order using complete, request-eligible native blocks. If none suffices within the cap, borrowing is rejected. The 94/95/96-frame controls vary static KV/expert allocation, not the borrowing cap. The cap is a fixed policy budget. Performance evidence covers finite traces with 128-token outputs.

Thank you for the concrete request. Rebinding enables temporary Expert/KV capacity sharing within a running engine. We connect this capability to reduced preemptions and report all 20 runs across repeated ablations and equal-budget fixed partitions.

W1 / Q1: Rebinding Value and Complete Controls

(1) Technical value: reversible capacity sharing. Rebinding enables reversible capacity sharing inside a running engine: selected expert backing is temporarily converted into request-owned KV capacity, while native expert tensors and KV block tables keep their existing interfaces. To make repeated conversions safe, ownership generations distinguish successive uses, and a block is allocatable only when all its backing is ready and authorized. After the loan's KV references are released, the backing can return to expert service. The Expert/KV partition of a fixed memory budget can therefore be adjusted over request lifetimes rather than fixed at startup. The zero-loan short workload illustrates the complementary roles: expert supply supports baseline serving, while rebinding addresses KV-pressure episodes.

(2) All repetitions. A/B denote Mixed 1/2, each with two process-level repetitions. R is ReFrame, Off retains the same configuration and expert-supply path with borrowing disabled, and Static is the fixed-pool control. Each run completes 36 requests and 4,608 output tokens. We report descriptive summaries of these repetitions.

RunMethodTTFT p50/p95 (s)TPOT p50/p95 (ms/token)Completion (s)
A1Off249.63/593.482223.28/2889.041342.02
A1R190.40/580.902011.73/2892.971290.87
A1Static284.82/638.682218.55/3180.881388.58
A2Off264.18/619.082239.24/3206.961375.78
A2R189.16/587.112083.80/2864.261299.46
A2Static240.38/587.922202.34/2883.821330.70
B1Off114.24/459.502059.07/3139.911295.77
B1R71.60/467.811916.16/3407.531314.91
B1Static95.56/443.421948.36/2966.781252.40
B2Off120.44/457.162166.63/3029.231275.90
B2R92.81/487.882358.35/3354.731330.11
B2Static129.59/461.672221.86/3178.601299.93
Four-pair geometric meanLong TTFTTPOT p95Completion
R/Off0.98081.01820.9901
R/Static0.98761.02290.9938

Across all four same-configuration pairs, native preemptions fall from 6/6/5/4 to 2/2/2/1, with zero loan-owner preemptions and restored expert capacity in every dynamic run. Relative to Off, the geometric mean gives 1.92% lower long TTFT and 0.99% lower completion time, alongside 1.82% higher TPOT p95. The complete pairs capture the combined effect on admission and subsequent service across workloads. Figure 8's 12.17% long-TTFT reduction corresponds to the retrospectively selected A2 case, with all repetitions now reported above.

(3) Equal-budget static partitions. The submitted ReFrame starts from K94. The fixed-partition results identify K96 as the strongest static starting point for TTFT on both traces: it has the lowest TTFT p50 and p95 among K94, K95, and K96. We therefore reran ReFrame from K96 (R96). All runs use a 1,974-frame physical arena per rank. R96 was measured in a separate campaign.

RunMethodTTFT p50/p95 (s)content-span/127 p50/p95 (ms)Completion (s)
AK94281.47/677.032137.88/2987.931404.96
AK95247.92/672.101989.50/3037.791328.94
AK96239.54/662.922110.92/3190.991332.53
AR96173.86/564.852097.50/2791.021297.04
BK94131.68/505.672431.24/2996.401327.08
BK95107.07/501.352391.74/3305.921350.95
BK9688.69/497.552316.35/3161.361323.95
BR9657.93/451.551908.99/3310.171268.86

Both R96 runs complete all 36 requests without runtime borrowing. Their results therefore describe the K96 starting configuration. Rebinding can adjust the Expert/KV partition during pressure and return the borrowed capacity to expert service.

W2 / Q2: Comparison with MoE-Specific Offloading Systems

We detail the adaptation, measurement protocol, results, and scope caveats for the MoE-Infinity comparison.

MoE-Infinity comparison. The adaptation enables MoE-Infinity to serve Qwen3-235B with expert CPU offloading. It includes changes to expert eviction, scheduling, stream initialization, and last-position logit projection. Six distinct runs completed 36 requests each, with 128 output tokens per request, for 216 requests and 27,648 tokens in total. The six four-GPU runs used 132.53 GPU-hours, including startup and shutdown but excluding failed attempts and development diagnostics.

All latencies are in seconds, ordered MoE-Infinity / ReFrame. GPU memory utilization: 0.98/0.98. The last column gives the completion-time ratio.

WorkloadMoE-Infinity requestsCompletion time (s)TTFT p95 (s)TPOT p95 (s)Completion ratio
Mixed 136/3619,223.58 / 1,281.426,057.97 / 578.7596.83 / 2.9315.00×
Mixed 236/3622,188.08 / 1,247.787,494.04 / 434.12108.36 / 3.1917.78×
Short requests36/3612,814.18 / 1,299.652,433.89 / 483.9982.81 / 3.449.86×
SummScreenFD + WildChat36/3620,374.85 / 1,102.066,740.52 / 412.76100.66 / 2.7518.49×
Arrival density 0.5×36/3620,584.89 / 1,749.546,176.58 / 483.43104.19 / 2.4011.77×
Arrival density 1׆36/3619,223.58 / 1,303.886,057.97 / 593.2096.83 / 3.1114.74×
Arrival density 2×36/3621,319.75 / 1,188.006,495.36 / 711.77110.98 / 3.2117.95×

† Density 1× reuses the Mixed 1 MoE-Infinity run with a separate historical ReFrame reference.

From unrounded data, mixed-workload geometric-mean MoE-Infinity/ReFrame ratios are 16.33× for completion, 13.44× for TTFT p95 and 33.51× for TPOT p95.

MoE-Infinity uses activation-aware expert caching and prefetching to reduce offloading costs. ReFrame coordinates expert residency and KV admission under a shared memory budget. For Qwen3-235B in BF16, overlapping requests place both expert supply and KV capacity under pressure on four 80 GB GPUs. ReFrame’s joint management is intended to address this competition and may help explain the observed difference.

Thanks for the reply. I will keep my positive score.

Reviewer_JmoEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets mixture-of-experts serving when weights exceed GPU memory and engines statically split the remaining memory between resident experts and the KV cache. ReFrame places both in one arena of 36 MiB typed frames with native logical views and a generation-tagged ownership protocol, while a bounded, request-isolated policy lends at most two frames per busy episode and allocates private blocks first. It builds on vLLM 0.11.0 with a rank-global expert cache and four host preparation workers. On Qwen3-235B-A22B with four A100 GPUs and two mixed workloads, it reports 2.75× and 2.94× geometric-mean completion speedups over vLLM 0.29 and SGLang 0.5.17 with CPU offload.

Critical strengths
  1. End-to-end gains over framework CPU offload persist across workloads, reruns, and arrival densities.

  2. Typed frames with generation-tagged ownership, a proven invariant, and audited frame return.

  3. Unusually transparent reporting of retrospective selection, single runs, and the weak four-pair average.

Critical weaknesses
  1. Section 5.4 candidly reports the all-pairs result: across all four Static ReFrame pairs, long-request TTFT improves by only 1.24% while TPOT p95 regresses by 2.29%, and Figure 8's 12% gains are retrospectively selected. Rebinding does cut native preemptions (Appendix G.1), yet ReFrame records zero loans on the short-request workload and is still 1.54× faster than vLLM 0.29 (Table 5). Each activation lends at most two frames (Equation 2), the selected run records one loan (Table 14), and one frame adds about 1% to the 94-frame KV pool, so a slightly larger static pool might suffice. Please report all repetitions and a static-pool sweep.

  2. Section 1 cites FluxMoE as dynamically reassigning GPU memory, and Section 2 notes that Oneiros repurposes parameter memory, yet the paper distinguishes itself from both only briefly and compares against neither. MoE-aware offloading systems discussed in Section 2, such as MoE-Infinity and Fiddler, are also not evaluated. All baselines use 40 GiB/GPU generic CPU offload (Section 5.1), and the 4.6× to 4.7× gap between vLLM 0.11 and vLLM 0.29 (Table 2) suggests that offload implementation dominates these comparisons. Please compare against FluxMoE or Oneiros and one MoE-aware offloading system at equal memory.

  3. Proposition 1 proves an ownership invariant (Appendix I.2), and Section 3.1 says expert slots keep kernel-compatible tensor layouts. However, the paper never explains how 36 MiB frames back KV tensors and expert slots, although blocks may span frames (Appendix I.3), how in-flight kernels are quiesced, or how the four ranks agree on conversions. Evidence of safety is limited to frame-count audits (Appendix G.1), and greedy outputs are never compared with the no-rebinding control. For a mechanism that aliases weight and KV memory, output correctness remains unverified. Please describe the mapping and synchronization, and report output agreement.

Questions for authors
  1. Averaged over all repetitions, how do ReFrame without rebinding and a 95- or 96-frame static pool compare with ReFrame?

  2. How does ReFrame compare with FluxMoE or Oneiros, and with an MoE-aware offloading system, under the same memory budget?

  3. How do frames map into KV and expert tensors, and do greedy outputs match the no-rebinding run?

3 comments

We thank the reviewer for these concrete requests, and we have completed the revisions addressing the second and third issues.

W2 / Q2: Comparison with Oneiros

We detail the adaptation, measurement protocol, results, and scope caveats for the Oneiros comparison.

Oneiros comparison. The adaptation enables Oneiros's parameter/KV remapping and restoration paths. It is based on Oneiros's vLLM 0.7.3 fork. Six distinct runs completed 36 requests each, with 128 output tokens per request, for 216 requests and 27,648 tokens in total. The six four-GPU runs used 97.87 GPU-hours, including startup and shutdown but excluding failed attempts and development diagnostics.

All latencies are in seconds, ordered Oneiros / ReFrame. GPU memory utilization: 0.98/0.98. The last column gives the completion-time ratio.

WorkloadCompleted requestsCompletion time (s)TTFT p95 (s)TPOT p95 (s)Completion ratio
Mixed 136/3613,156.71 / 1,281.429,933.88 / 578.7521.50 / 2.9310.27×
Mixed 236/3612,118.22 / 1,247.789,126.46 / 434.1220.25 / 3.199.71×
Short requests36/3611,331.21 / 1,299.658,703.89 / 483.9918.30 / 3.448.72×
SummScreenFD + WildChat36/3611,957.31 / 1,102.069,044.55 / 412.7619.98 / 2.7510.85×
Arrival density 0.5×36/3612,084.81 / 1,749.548,349.80 / 483.4319.64 / 2.406.91×
Arrival density 1׆36/3613,156.71 / 1,303.889,933.88 / 593.2021.50 / 3.1110.09×
Arrival density 2×36/3611,866.63 / 1,188.009,285.99 / 711.7719.94 / 3.219.99×

† Density 1× reuses the Mixed 1 Oneiros run with a separate historical ReFrame reference.

From unrounded data, mixed-workload geometric-mean Oneiros/ReFrame ratios are 9.99× for completion, 19.00× for TTFT p95 and 6.83× for the TPOT p95.

Oneiros's published evaluation targets multi-tenant GH200 serving. Our experiments use an oversized BF16 MoE on four 80 GB A100s, with different transfer/computation costs.

W3 / Q3: Frame Mapping, Synchronization and Output Checks

We clarify three aspects: native tensor mapping, CUDA and rank coordination, and output agreement.

(1) Native tensor mapping. In the submitted BF16 layout, a 36 MiB frame holds w13=[3072,4096] (24 MiB) followed by w2=[4096,1536] (12 MiB), at fixed offsets with no padding.

The KV view for each layer is [2,6912,16,1,128], ordered as K/V, block, token, head and head dimension. A logical block occupies B=770,048 bytes across 94 layers. Block b occupies [bB,(b+1)B) and is backed by every frame intersecting that interval. A free block is allocatable only when all backing frames are ready and authorized for its request. Request-scoped loans reserve capacity for their owner. Private-first allocation uses eligible private blocks before shared blocks. The 6912 blocks specify the virtual capacity. Their availability depends on the backing frames.

(2) CUDA and rank coordination. Conversion first quiesces the outgoing owner and retires its publication. It then materializes the new role and publishes a new generation. Attention writes KV into the available backing. Returning a frame to Expert ownership restores its weights before publication. Each rank waits on explicit CUDA stream events and incoming materialization. Native scheduling supplies common logical block assignments, while physical handles and publications stay local. Proposition 1 establishes frame-local protection under eager execution, serialized rank entry and complete producer/consumer stream coverage.

(3) Output agreement. A controlled four-rank repayment test matched the token outputs of 10/10 requests against its reference. It also verified that the restored expert weights matched their host copies on every rank while the relocated KV state remained live. These checks directly examined generated tokens and restored tensor contents beyond frame-count audits. The test used development code, and its reference also allowed borrowing with a different driver. As a complementary repeatability check, a development shadow-mode A/A test with no borrowing produced 480/480 byte-identical full-vocabulary logit vectors across four ranks. These results support numerical repeatability in the zero-borrowing configuration and output agreement in the controlled repayment test.

Thank you for the concrete request. Rebinding enables temporary Expert/KV capacity sharing within a running engine. We connect this capability to reduced preemptions and report all 20 runs across repeated ablations and equal-budget fixed partitions.

W1 / Q1: Rebinding Value and Complete Controls

(1) Technical value: reversible capacity sharing. Rebinding enables reversible capacity sharing inside a running engine: selected expert backing is temporarily converted into request-owned KV capacity, while native expert tensors and KV block tables keep their existing interfaces. To make repeated conversions safe, ownership generations distinguish successive uses, and a block is allocatable only when all its backing is ready and authorized. After the loan's KV references are released, the backing can return to expert service. The Expert/KV partition of a fixed memory budget can therefore be adjusted over request lifetimes rather than fixed at startup. The zero-loan short workload illustrates the complementary roles: expert supply supports baseline serving, while rebinding addresses KV-pressure episodes.

(2) All repetitions. A/B denote Mixed 1/2, each with two process-level repetitions. R is ReFrame, Off retains the same configuration and expert-supply path with borrowing disabled, and Static is the fixed-pool control. Each run completes 36 requests and 4,608 output tokens. We report descriptive summaries of these repetitions.

RunMethodTTFT p50/p95 (s)TPOT p50/p95 (ms/token)Completion (s)
A1Off249.63/593.482223.28/2889.041342.02
A1R190.40/580.902011.73/2892.971290.87
A1Static284.82/638.682218.55/3180.881388.58
A2Off264.18/619.082239.24/3206.961375.78
A2R189.16/587.112083.80/2864.261299.46
A2Static240.38/587.922202.34/2883.821330.70
B1Off114.24/459.502059.07/3139.911295.77
B1R71.60/467.811916.16/3407.531314.91
B1Static95.56/443.421948.36/2966.781252.40
B2Off120.44/457.162166.63/3029.231275.90
B2R92.81/487.882358.35/3354.731330.11
B2Static129.59/461.672221.86/3178.601299.93
Four-pair geometric meanLong TTFTTPOT p95Completion
R/Off0.98081.01820.9901
R/Static0.98761.02290.9938

Across all four same-configuration pairs, native preemptions fall from 6/6/5/4 to 2/2/2/1, with zero loan-owner preemptions and restored expert capacity in every dynamic run. Relative to Off, the geometric mean gives 1.92% lower long TTFT and 0.99% lower completion time, alongside 1.82% higher TPOT p95. The complete pairs capture the combined effect on admission and subsequent service across workloads. Figure 8's 12.17% long-TTFT reduction corresponds to the retrospectively selected A2 case, with all repetitions now reported above.

(3) Equal-budget static partitions. The submitted ReFrame starts from K94. The fixed-partition results identify K96 as the strongest static starting point for TTFT on both traces: it has the lowest TTFT p50 and p95 among K94, K95, and K96. We therefore reran ReFrame from K96 (R96). All runs use a 1,974-frame physical arena per rank. R96 was measured in a separate campaign.

RunMethodTTFT p50/p95 (s)content-span/127 p50/p95 (ms)Completion (s)
AK94281.47/677.032137.88/2987.931404.96
AK95247.92/672.101989.50/3037.791328.94
AK96239.54/662.922110.92/3190.991332.53
AR96173.86/564.852097.50/2791.021297.04
BK94131.68/505.672431.24/2996.401327.08
BK95107.07/501.352391.74/3305.921350.95
BK9688.69/497.552316.35/3161.361323.95
BR9657.93/451.551908.99/3310.171268.86

Both R96 runs complete all 36 requests without runtime borrowing. Their results therefore describe the K96 starting configuration. Rebinding can adjust the Expert/KV partition during pressure and return the borrowed capacity to expert service.

W2 / Q2: Comparison with MoE-Specific Offloading Systems

We detail the adaptation, measurement protocol, results, and scope caveats for the MoE-Infinity comparison.

MoE-Infinity comparison. The adaptation enables MoE-Infinity to serve Qwen3-235B with expert CPU offloading. It includes changes to expert eviction, scheduling, stream initialization, and last-position logit projection. Six distinct runs completed 36 requests each, with 128 output tokens per request, for 216 requests and 27,648 tokens in total. The six four-GPU runs used 132.53 GPU-hours, including startup and shutdown but excluding failed attempts and development diagnostics.

All latencies are in seconds, ordered MoE-Infinity / ReFrame. GPU memory utilization: 0.98/0.98. The last column gives the completion-time ratio.

WorkloadMoE-Infinity requestsCompletion time (s)TTFT p95 (s)TPOT p95 (s)Completion ratio
Mixed 136/3619,223.58 / 1,281.426,057.97 / 578.7596.83 / 2.9315.00×
Mixed 236/3622,188.08 / 1,247.787,494.04 / 434.12108.36 / 3.1917.78×
Short requests36/3612,814.18 / 1,299.652,433.89 / 483.9982.81 / 3.449.86×
SummScreenFD + WildChat36/3620,374.85 / 1,102.066,740.52 / 412.76100.66 / 2.7518.49×
Arrival density 0.5×36/3620,584.89 / 1,749.546,176.58 / 483.43104.19 / 2.4011.77×
Arrival density 1׆36/3619,223.58 / 1,303.886,057.97 / 593.2096.83 / 3.1114.74×
Arrival density 2×36/3621,319.75 / 1,188.006,495.36 / 711.77110.98 / 3.2117.95×

† Density 1× reuses the Mixed 1 MoE-Infinity run with a separate historical ReFrame reference.

From unrounded data, mixed-workload geometric-mean MoE-Infinity/ReFrame ratios are 16.33× for completion, 13.44× for TTFT p95 and 33.51× for TPOT p95.

MoE-Infinity uses activation-aware expert caching and prefetching to reduce offloading costs. ReFrame coordinates expert residency and KV admission under a shared memory budget. For Qwen3-235B in BF16, overlapping requests place both expert supply and KV capacity under pressure on four 80 GB GPUs. ReFrame’s joint management is intended to address this competition and may help explain the observed difference.

Comments on the paper

3 comments

Dear Area Chair,

We thank you and the reviewers for the careful evaluation and constructive discussion. The added serving measurements show faster trace completion for ReFrame than for the adapted Oneiros and MoE-Infinity baselines under the reported configurations. Matched controls show fewer preemptions with rebinding and restoration of borrowed expert capacity.

1. Reviewer JmoE: Contribution, baselines, and correctness.

  • W1/Q1: Rebinding provides reversible KV capacity with fewer preemptions in the matched controls. The complete borrowing-on/off comparisons record fewer native preemptions, no loan-owner preemption, and restored expert capacity in every dynamic run. The partition sweep identifies K96 as the strongest static starting point for TTFT. Starting from K96, ReFrame outperforms static K96 in both TTFT p95 and trace completion under the same physical memory budget. TTFT p95 decreases from 662.92/497.55 s to 564.85/451.55 s, and completion time decreases from 1,332.53/1,323.95 s to 1,297.04/1,268.86 s. The matched borrowing-on/off controls above separately quantify rebinding's incremental effect.
  • W2/Q2: ReFrame's measured completion advantage extends to both requested baseline families. The Oneiros and MoE-Infinity experiments cover parameter/KV remapping and specialized MoE offloading across the main-text workloads. ReFrame completes every reported trace sooner in these system-level measurements, extending the evidence beyond generic framework offloading.
  • W3/Q3: Controlled tests support expert restoration while KV remains live. The development repayment test reproduces its reference token outputs and verifies restored expert weights on every rank. The zero-borrowing A/A logit test supports repeatability in its tested configuration. The tensor/block mappings and CUDA-event coordination explain how access is restricted to ready, authorized backing under the stated synchronization assumptions.

2. Reviewer Q1m4: Attribution, allocation, and operating boundaries.

  • W1/Q1: Rebinding has a measurable incremental effect beyond expert supply. With the configuration and expert-supply path held fixed, the complete repeated controls show lower aggregate long-request TTFT and trace completion, alongside a decoding-latency trade-off. Preemption and restoration records support temporary pressure relief followed by return to expert service.
  • W2/Q2: ReFrame makes reclaimed expert storage usable by a specific request. A KV block is eligible only when every backing frame is ready and authorized, including blocks crossing frame boundaries. Ownership protection and rank-local publication enforce this condition. This connects memory remapping to usable KV capacity and request admission. Cross-layout diagnostics also show reference greedy-token agreement for ReFrame, providing a token-level check.
  • W2/Q3: The policy bounds borrowing while preserving live request state. A busy episode resets only when both active and pending sets are empty, preventing repeated activation under continuous arrivals. During long decoding, existing loans remain available until their live KV references are released. The threshold governs activation, and the frame cap limits displaced expert capacity. Existing sensitivity results characterize these choices in the tested regime. Native scheduling handles remaining shortages.

3. Reviewer 2Qnc: Relevance, added capabilities, and MoE-specific comparisons.

  • W1: ReFrame addresses a foundation-model inference bottleneck. Expert weights and growing KV state compete for GPU capacity in large MoE serving. ReFrame coordinates this capacity between expert execution and request admission, directly addressing the selected primary area, "foundation or frontier models, including LLMs."
  • W2: ReFrame adds expert-level reallocation with a request-specific lifetime. It reclaims selected frames within an active MoE layer and reserves them for the triggering request. After that request releases its live KV state, the frames become eligible for expert restoration while other requests continue. This adds finer allocation and lifetime control to Oneiros's published model/layer policy and addresses expert-parameter/KV sharing beyond eLLM's activation/KV sharing.
  • W3: The MoE-specific comparison supports ReFrame's serving advantage in the reported setting. The reported MoE-Infinity measurements show lower TTFT p95, TPOT p95, and trace completion for ReFrame across all evaluated workloads, with mixed-workload geometric-mean MoE-Infinity/ReFrame ratios of 13.44×, 33.51×, and 16.33×, respectively. Expert caching and prefetching optimize weight delivery, while ReFrame also coordinates residency with KV admission.

We have also improved the manuscript's presentation, added experiments into the main text and appendix, and uploaded the revised PDF. We hope these results provide a clear basis for your assessment.

Additional comparisons with Oneiros and MoE-Infinity

Adaptation. We add Oneiros and MoE-Infinity to cover parameter/KV remapping and MoE-specific offloading, respectively. The Oneiros adaptation is based on its vLLM 0.7.3 fork and enables parameter/KV remapping and restoration. The MoE-Infinity adaptation serves Qwen3-235B with expert CPU offloading, with changes to expert eviction, scheduling, stream initialization, and last-position logit projection.

Measurement protocol. Each baseline completes six distinct runs of 36 requests and 128 output tokens per request, totaling 216 requests and 27,648 output tokens. The successful four-GPU runs use 97.87 GPU-hours for Oneiros and 132.53 GPU-hours for MoE-Infinity, including startup and shutdown but excluding failed attempts and development diagnostics.

Table1. Oneiros comparison across the main-text workloads. Entries are Oneiros/ReFrame, and the last column is the completion-time ratio. Every run completes 36 requests with 128 output tokens per request.

WorkloadCompletion (s)TTFT p95 (s)TPOT p95 (s)Ratio
Mixed 113,156.71 / 1,281.429,933.88 / 578.7521.50 / 2.9310.27×
Mixed 212,118.22 / 1,247.789,126.46 / 434.1220.25 / 3.199.71×
Short requests11,331.21 / 1,299.658,703.89 / 483.9918.30 / 3.448.72×
SummScreenFD + WildChat11,957.31 / 1,102.069,044.55 / 412.7619.98 / 2.7510.85×
Density 0.5×12,084.81 / 1,749.548,349.80 / 483.4319.64 / 2.406.91×
Density 1׆13,156.71 / 1,303.889,933.88 / 593.2021.50 / 3.1110.09×
Density 2×11,866.63 / 1,188.009,285.99 / 711.7719.94 / 3.219.99×

Table2. MoE-Infinity comparison across the main-text workloads. Entries are MoE-Infinity/ReFrame, and the last column is the completion-time ratio. Every run completes 36 requests with 128 output tokens per request.

WorkloadCompletion (s)TTFT p95 (s)TPOT p95 (s)Ratio
Mixed 119,223.58 / 1,281.426,057.97 / 578.7596.83 / 2.9315.00×
Mixed 222,188.08 / 1,247.787,494.04 / 434.12108.36 / 3.1917.78×
Short requests12,814.18 / 1,299.652,433.89 / 483.9982.81 / 3.449.86×
SummScreenFD + WildChat20,374.85 / 1,102.066,740.52 / 412.76100.66 / 2.7518.49×
Density 0.5×20,584.89 / 1,749.546,176.58 / 483.43104.19 / 2.4011.77×
Density 1׆19,223.58 / 1,303.886,057.97 / 593.2096.83 / 3.1114.74×
Density 2×21,319.75 / 1,188.006,495.36 / 711.77110.98 / 3.2117.95×

For both tables, density 1× (†) reuses the Mixed 1 baseline run with a separate historical ReFrame reference. From unrounded data, the mixed-workload geometric-mean baseline/ReFrame ratios for completion, TTFT p95, and TPOT p95 are 9.99×, 19.00×, and 6.83× for Oneiros, and 16.33×, 13.44×, and 33.51× for MoE-Infinity.

Complete Rebinding Controls and Runtime Evidence

All repeated ablations. Table1 reports all repetitions of the rebinding ablation. A/B denote Mixed workloads 1/2, each with two process-level repetitions. ReFrame without rebinding retains the same configuration and expert-supply path with borrowing disabled. Static ReFrame is the fixed-pool control. Each run completes 36 requests and 4,608 output tokens. TPOT denotes the content-span/127 statistic.

Table 1. All repetitions of the original K94 ablation. TTFT and completion are in seconds; TPOT is in seconds per token.

RunMethodTTFT p50/p95 (s)TPOT p50/p95 (s)Completion (s)
A1ReFrame without rebinding249.63/593.482.22/2.891342.02
A1ReFrame190.40/580.902.01/2.891290.87
A1Static ReFrame284.82/638.682.22/3.181388.58
A2ReFrame without rebinding264.18/619.082.24/3.211375.78
A2ReFrame189.16/587.112.08/2.861299.46
A2Static ReFrame240.38/587.922.20/2.881330.70
B1ReFrame without rebinding114.24/459.502.06/3.141295.77
B1ReFrame71.60/467.811.92/3.411314.91
B1Static ReFrame95.56/443.421.95/2.971252.40
B2ReFrame without rebinding120.44/457.162.17/3.031275.90
B2ReFrame92.81/487.882.36/3.351330.11
B2Static ReFrame129.59/461.672.22/3.181299.93

Across all four same-configuration pairs, native preemptions fall from 6/6/5/4 to 2/2/2/1, with zero loan-owner preemptions and restored expert capacity in every dynamic run. Relative to Off, the geometric mean gives 1.92% lower long TTFT and 0.99% lower completion time, alongside 1.82% higher TPOT p95. The complete pairs capture the combined effect on admission and subsequent service across workloads.

Equal-budget static partitions. The submitted ReFrame starts from K94. The K94/K95/K96 comparison uses one run per method per workload, with the same audited 1,974-frame physical arena per rank. K95 and K96 exchange one and two expert frames for KV at startup, respectively. K96 has the lowest static TTFT p50 and p95 on both traces. We therefore reran ReFrame from K96, shown as ReFrame (K96 start) in Table 2.

Table 2. Equal-budget fixed partitions and ReFrame starting from K96. A/B denote Mixed workloads 1/2. The ReFrame runs from K96 were measured in a separate campaign. Each row completes 36 requests and 4,608 output tokens.

RunMethodTTFT p50/p95 (s)TPOT p50/p95 (s)Completion (s)
AK94281.47/677.032.14/2.991404.96
AK95247.92/672.101.99/3.041328.94
AK96239.54/662.922.11/3.191332.53
AReFrame (K96 start)173.86/564.852.10/2.791297.04
BK94131.68/505.672.43/2.991327.08
BK95107.07/501.352.39/3.311350.95
BK9688.69/497.552.32/3.161323.95
BReFrame (K96 start)57.93/451.551.91/3.311268.86

Both R96 runs complete all 36 requests without runtime borrowing. Their results therefore describe the K96 starting configuration. Rebinding can adjust the Expert/KV partition during pressure and return the borrowed capacity to expert service.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.