- Rating
- 2
- Confidence
- 1
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper presents ReFrame, an MoE serving runtime that dynamically shares GPU memory between expert weights and KV cache through typed frame rebinding and a bounded, request-isolated borrowing policy. Implemented on vLLM and evaluated on Qwen3-235B-A22B, it reports 2.75× and 2.94× completion-time speedups over vLLM and SGLang with CPU offloading.
Critical strengths
A typed frame rebinding mechanism that allows GPU memory to switch between expert weights and KV cache while keeping native tensor layouts and KV block tables unchanged, supported by an ownership protocol that prevents frames in use from being reclaimed.
A bounded, request-isolated borrowing policy that borrows only the smallest feasible number of frames, reserves them for the triggering request, and uses private blocks first to preserve shared capacity.
A full implementation on vLLM with a careful experimental protocol, including fresh processes for each run, reversed execution order, and frozen inputs, evaluated on a 235B-parameter MoE model.
Critical weaknesses
This is a pure systems paper with no learning or representation contribution, and it would be a better fit for venues such as OSDI, SOSP, EuroSys or MLSys.
The novelty is limited, since eLLM already supports elastic memory sharing between runtime tensors and KV cache, and Oneiros already remaps parameter memory to KV cache. The paper should clarify what is technically new beyond applying this idea to MoE expert weights.
All baselines use generic layer-wise CPU offloading, which transfers weights to the GPU on every forward pass, while ReFrame uses an MoE-aware expert cache. The paper should compare against MoE-specific offloading systems.
Questions for authors
See the above weaknesses.
3 comments
W2: Part (1) of the response restates the mechanisms already described in the paper (ownership transitions, request-scoped loans, private-first allocation, and a prefix capped at two frames), but it does not identify what ReFrame can do that Oneiros cannot.
W3: The response does not address this point. The comparison against MoE-specific offloading systems such as MoE-Infinity, Fiddler, ProMoE or MoE-Lightning is still missing. This concern remains unresolved.
Thanks for your valuable suggestion, and we have completed the revisions addressing the second and third questions.
W2: Capability beyond Oneiros
ReFrame lets individual requests borrow expert frames for KV capacity while the MoE model continues serving. Oneiros already supports reversible parameter/KV remapping, including for active models. Its controller selects models and layers to remap, and its transfer engine uses the recurring order of layer execution. ReFrame extends this form of memory sharing with expert-level reallocation and request-specific allocation rules.
(1) Reclaim part of an active MoE layer. In our Qwen3-235B layout, one 36 MiB frame holds one expert's gate/up and down weights. When a request lacks KV blocks, ReFrame can lend eligible expert frames without remapping the remaining experts in those layers. The rank-wide expert cache and host-backed loading path continue to supply routed experts as residency changes. ReFrame can therefore adjust residency within a layer, whereas Oneiros's published policy reallocates capacity at layer granularity.
(2) Reserve reclaimed capacity for the request that needs it. Consider requests A and B sharing the same KV pool. If A triggers borrowing, B cannot allocate KV blocks backed by A's borrowed frames. A can use a block only when every backing frame is ready and authorized, even when the block spans multiple frames. After A releases the last live KV references, those frames become eligible for expert reconstruction without waiting for B to finish. The loan's lifetime is thus tied to A's KV use. Oneiros's published model/layer controller does not provide this request-owned expert-frame allocation rule.
The two-frame cap bounds the evaluated policy. ReFrame's added capability is to connect expert-level memory reallocation to a particular request's admission and KV lifetime. This gives the runtime control over which expert capacity is reassigned, which request can use it, and when it can return to expert residency.
W3: Comparison with MoE-Specific Offloading Systems
We detail the adaptation, measurement protocol, results, and scope caveats for the MoE-Infinity comparison.
MoE-Infinity comparison. The adaptation enables MoE-Infinity to serve Qwen3-235B with expert CPU offloading. It includes changes to expert eviction, scheduling, stream initialization, and last-position logit projection. Six distinct runs completed 36 requests each, with 128 output tokens per request, for 216 requests and 27,648 tokens in total. The six four-GPU runs used 132.53 GPU-hours, including startup and shutdown but excluding failed attempts and development diagnostics.
All latencies are in seconds, ordered MoE-Infinity / ReFrame. GPU memory utilization: 0.98/0.98. The last column gives the completion-time ratio.
| Workload | MoE-Infinity requests | Completion time (s) | TTFT p95 (s) | TPOT p95 (s) | Completion ratio |
|---|---|---|---|---|---|
| Mixed 1 | 36/36 | 19,223.58 / 1,281.42 | 6,057.97 / 578.75 | 96.83 / 2.93 | 15.00× |
| Mixed 2 | 36/36 | 22,188.08 / 1,247.78 | 7,494.04 / 434.12 | 108.36 / 3.19 | 17.78× |
| Short requests | 36/36 | 12,814.18 / 1,299.65 | 2,433.89 / 483.99 | 82.81 / 3.44 | 9.86× |
| SummScreenFD + WildChat | 36/36 | 20,374.85 / 1,102.06 | 6,740.52 / 412.76 | 100.66 / 2.75 | 18.49× |
| Arrival density 0.5× | 36/36 | 20,584.89 / 1,749.54 | 6,176.58 / 483.43 | 104.19 / 2.40 | 11.77× |
| Arrival density 1׆ | 36/36 | 19,223.58 / 1,303.88 | 6,057.97 / 593.20 | 96.83 / 3.11 | 14.74× |
| Arrival density 2× | 36/36 | 21,319.75 / 1,188.00 | 6,495.36 / 711.77 | 110.98 / 3.21 | 17.95× |
† Density 1× reuses the Mixed 1 MoE-Infinity run with a separate historical ReFrame reference.
From unrounded data, mixed-workload geometric-mean MoE-Infinity/ReFrame ratios are 16.33× for completion, 13.44× for TTFT p95 and 33.51× for TPOT p95.
MoE-Infinity uses activation-aware expert caching and prefetching to reduce offloading costs. ReFrame coordinates expert residency and KV admission under a shared memory budget. For Qwen3-235B in BF16, overlapping requests place both expert supply and KV capacity under pressure on four 80 GB GPUs. ReFrame’s joint management is intended to address this competition and may help explain the observed difference.
We are glad to have clarified ReFrame's capabilities beyond Oneiros and strengthened the comparison with MoE-specific offloading systems through the added MoE-Infinity experiments. We hope our response is satisfactory.
We thank the reviewer for highlighting scope, novelty and baseline coverage, and we have completed the revisions addressing the first and second issues.
W1: Venue Fit and Relevance to ICLR
ReFrame studies inference for a 235B MoE foundation model whose BF16 weights exceed four A100 GPUs' aggregate memory. Its ownership and allocation rules address competition between reconstructible expert weights and growing request-specific KV state. This contribution to foundation-model inference fits the selected primary area and aligns with the ICLR call.
W2: Novelty Beyond eLLM and Oneiros
We clarify novelty beyond eLLM and Oneiros in three parts: allocation semantics for expert/KV reuse, choice of experimental baseline, and the added Oneiros comparison.
(1) Allocation semantics for expert/KV reuse. A free native KV block is eligible only when every backing frame is ready and authorized for its request, including blocks that cross frame boundaries. Conversion first quiesces the outgoing owner and retires its publication. It then materializes the new role and publishes a new generation. Active KV stays protected. Request-scoped loans reserve temporary capacity for their owner. At a fixed state with feasible demand and independently selectable blocks, private-first minimizes shared-block consumption. The controller selects the smallest feasible prefix of a fixed candidate order, capped at two frames, using complete native blocks. These rules connect physical reuse to native tensor views and allocation decisions. The implementation uses explicit stream events assuming eager execution, serialized rank entry and complete producer/consumer stream coverage.
(2) Choice of experimental baseline. eLLM shares physical memory between activation buffers and KV tensors and supports KV offloading to CPU memory. Oneiros directly remaps parameter backing into KV capacity and thus more closely matches ReFrame's expert-to-KV reuse. Evaluating this mechanism with eLLM would require extending its published design to reclaim expert parameter memory. The additional measurements therefore focus on Oneiros. eLLM remains relevant to the broader discussion of elastic memory management.
(3) Added Oneiros comparison. The adaptation enables Oneiros's parameter/KV remapping and restoration paths. It is based on Oneiros's vLLM 0.7.3 fork. Six distinct runs completed 36 requests each, with 128 output tokens per request, for 216 requests and 27,648 tokens in total. The six four-GPU runs used 97.87 GPU-hours, including startup and shutdown but excluding failed attempts and development diagnostics.
All latencies are in seconds, ordered Oneiros / ReFrame. GPU memory utilization: 0.98/0.98. The last column gives the completion-time ratio.
Density 1× reuses the Mixed 1 Oneiros run with a separate historical ReFrame reference.
From unrounded data, mixed-workload geometric-mean Oneiros/ReFrame ratios are 9.99× for completion, 19.00× for TTFT p95 and 6.83× for the content-span proxy p95.
Oneiros's published evaluation targets multi-tenant GH200 serving. Our experiments use an oversized BF16 MoE on four 80 GB A100s, with different transfer/computation costs.