- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents MIA, a plugin for vLLM that supports request-specific capture and steering of model internals while preserving full CUDA Graph replay. Its Aperture mechanism represents changing access requirements as routing entries consumed by operators installed before graph capture. A Scheduler maps request selections to the current batch and assigns capture destinations, while a Courier asynchronously offloads and reconstructs captured artifacts before releasing their storage. Experiments evaluate hidden-state capture, query/key capture, and activation steering under open-loop serving workloads, including heterogeneous layer selections and a four-GPU deployment of Llama-3.1-70B. The results show low median TPOT overhead and substantial improvements over callback-based access implementations.
Critical strengths
A coherent design for fine-grained access under optimized serving. Expressing request-specific selections and capture destinations as routing data provides a practical way to accommodate changing batches without inserting Python callbacks into graph replay. The design also explicitly addresses padded lanes, synchronization, request ownership, and delayed page reuse. These details matter because the usefulness of the interface depends on preserving both execution efficiency and correct data association.
Promising performance with meaningful experimental controls. MIA achieves low median TPOT overhead across capture and steering workloads and matches native vLLM’s SLO thresholds at the sampled arrival rates. Open-loop arrivals, controlled steering output lengths, idle-versus-active measurements, and the 70B tensor-parallel experiment strengthen the evidence that the implementation can be useful beyond a simple offline demonstration.
Critical weaknesses
-
The technical contribution relative to the closest systems is not sufficiently isolated. DMI-Lib already combines graph-compatible capture operators, bounded GPU staging, asynchronous export, and runtime request selection. In particular, its Section 4.4 describes a batch-aligned keep/drop vector consumed by a GPU gather-compact operation without changing the captured graph. EasySteer v0.29.0 also documents heterogeneous per-request steering configurations and persistent in-graph payload and selection buffers. These capabilities do not establish equivalence to MIA’s heterogeneous capture-layer routing, but they narrow the novelty claim substantially. The strongest apparent distinction is coordinating fine-grained capture selections and reusable destinations without recapturing the graph. The paper should explain more precisely what additional mechanism makes this possible and distinguish its contribution from kernel fusion, batched transfers, and plugin integration. Existing microbenchmarks do not sufficiently isolate these contributions. This uncertainty is consequential because technical differentiation is central to my assessment. See DMI-Lib and the EasySteer v0.29.0 release description.
-
The evaluation does not fully validate the system’s central access and collection guarantees. The heterogeneous experiment varies captured layers, while the steering benchmark uses identical vectors and strengths across requests. The paper does not report systematic checks of captured-value accuracy, request/token alignment under changing batches, or isolation between steered and unsteered requests. These are central requirements of the proposed interface. Furthermore, the 30-second measurement windows provide limited evidence about sustained operation when collection becomes the bottleneck acknowledged in Appendix B. Without buffer-occupancy, collection-wait, and artifact-completion measurements, it is difficult to determine whether low token-generation overhead also implies a stable and timely internal-data service. These gaps limit confidence in the broader serving claims.
Questions for authors
-
Clarification of novelty: What specific mechanism is required beyond extending DMI-Lib’s runtime selection to finer granularity or EasySteer’s persistent selection buffers? Please distinguish capabilities unavailable in the evaluated baseline versions from implementation choices that primarily improve performance. Which baseline commits were used?
-
Clarification and validation of access semantics: Does one engine support independently varying layer, token, and steering selections across requests, including concurrent capture and steering? What correctness checks establish numerical agreement, request/token alignment, and isolation under batch changes and repeated page reuse? Existing validation results would materially strengthen my assessment.
-
Clarification of collection stability and latency accounting: Did the reported runs reach a stable capture-buffer occupancy and export rate? How are collection delays after the final generated token reflected in the reported metrics? Please provide available evidence on time waiting for free pages and the delay until a completed artifact becomes available.
5 comments
Response to Q2 and Weakness 2: Access semantics and correctness checks
Thank you for this question. Yes: in one MIA engine, each request independently chooses capture (hidden states or queries/keys) or steering, along with the layers, tokens and stages (prefill, decode or both) to which it applies, and for steering also its vector and strength. Capture and steering can also run concurrently in the same engine. Implementation details can viewed through the github repo in the paper.
Per reviewer's suggestion, our rebuttal adds a heterogeneous steering workload on Qwen2.5-1.5B-Instruct with ShareGPT prompts at 1 to 64 RPS. Each request draws its own setting: one of 16 vectors, a strength (2, 5 or 10), the layers (one, three or all 28), the tokens (all or only the last) and the phase (prefill, decode or both). One in five requests is not steered. Requests with different settings share each batch under CUDA graphs.
Median TPOT overhead over native vLLM 0.29 under heterogeneous steering (Qwen2.5-1.5B-Instruct):
| 1 RPS | 4 RPS | 8 RPS | 16 RPS | 32 RPS | 64 RPS | |
|---|---|---|---|---|---|---|
| MIA | +3.9 % | +3.9 % | +4.1 % | +5.7 % | +11.2 % | +16.9 % |
| EasySteer | +5.6 % | +5.7 % | +6.3 % | +9.2 % | +46.3 % | saturated |
| Hotwire-vllm | +6.1 % | +7.7 % | +10.2 % | +28.7 % | 3.7× | saturated |
| vLLM-Hook | 4.4× | 4.5× | 4.9× | 6.0× | saturated | – |
| vLLM-Lens | 8.4× | 43.1× | saturated | – | – | – |
Up to 16 RPS, heterogeneous steering costs MIA about as much as the uniform steering in Table 1 (+3.7 to 5.6 %). Beyond 16 RPS, the overhead rises to 11 % at 32 RPS and 17 % at 64 RPS. MIA still keeps up at 64 RPS. EasySteer and Hotwire-vllm also keep CUDA graphs. However, their overhead grows faster with load. Both fall behind at 64 RPS. vLLM-Hook and vLLM-Lens run in eager mode and reach 4.4× and 8.4× the native TPOT already at 1 RPS. All results and details are now in Appendix E.4.
Numerical Agreement
we compared MIA and native vLLM with Hugging Face Transformers (HF) in eager mode with bfloat16 precision under greedy decoding on ShareGPT prompts, capturing all 32 layers of Llama-3.1-8B and steering Qwen2.5-1.5B-Instruct.
Since HF and vLLM compute in bfloat16 with different kernels, native vLLM even differs from HF by 1.3 % in hidden states on a typical prompt and by up to 5.7 % on the worst one. This deviation sets the precision floor for any vLLM-based system. In eager mode, MIA's captured tensors and steered generations match with native vLLM's tensors. Under CUDA graphs, MIA's hidden states are remains close to native vLLM's hidden-state extraction. Its queries and keys also stay at the eager level. The tables below use HF in eager mode as the reference. Each cell reports the largest difference of native vLLM or MIA from HF. We did not collect queries/keys or evaluate steering for vLLM under CUDA graphs because it does not support these operations natively.
Largest relative difference of captured tensors from HF:
| Eager | CUDA graph | |||
|---|---|---|---|---|
| vLLM | MIA | vLLM | MIA | |
| Hidden states | 5.666 % | 5.666 % | 6.556 % | 6.558 % |
| Queries | 4.541 % | 4.541 % | n/a | 5.028 % |
| Keys | 4.137 % | 4.137 % | n/a | 4.783 % |
Since steering is for token generation, we compare the log-probability each system assigns to each generated token before the outputs first diverge.
Largest log-probability difference of steered tokens from HF:
| Eager | CUDA graph | |||
|---|---|---|---|---|
| vLLM | MIA | vLLM | MIA | |
| Steering | 0.186 nats | 0.186 nats | n/a | 0.198 nats |
Request and Token Alignment
We served requests with heterogeneous capture settings at 8 RPS. Buffer pages were reused several times on average during this workload. We then decoded each captured last-layer hidden state into a token, since a row associated with the wrong request or token position could decode to a different token. The decoded token matches the emitted token for 99.6 % of vectors, compared with 99.7 % for native vLLM running each request alone in eager mode.
Isolation under Batch Change
We replayed every request of a heterogeneous steering run (16 RPS, prefix caching off) alone, once with its own steering and once with a different vector. Of the 142 requests whose replays can be told apart, 141 follow their own steering, and unsteered requests drift no more than steered ones, so we find no sign of steering leaking across requests.
Appendix C.5 has the full correctness checks, and Appendix E.4 the heterogeneous steering results and the isolation check.
Response to Q3 and Weakness 2 (1/2): Buffer stability under collection pressure
To address your concern, we traced the 4 GiB paged capture buffer at every engine step on the Table 1 workloads.
4 GiB buffer at 16 RPS (Llama-3.1-8B, all 32 layers):
| Hidden states | Queries/keys | |
|---|---|---|
| Mean buffer occupancy (%) | 3.4% | 61.1% |
| Median page waits per request (times) | 0 | 0.09 |
| Median page wait latency (ms) | 0 ms | 6.4 ms |
| p99 page wait latency (ms) | 0 ms | 415 ms |
Hidden-state capture never waits for a page from 1 to 16 RPS, so buffer pressure does not limit its near-native cost. Query/key capture stores more data per token than hidden-state capture and takes longer to write out of the buffer. At high load, it therefore runs close to the buffer's limit. These waits are brief for most requests and mainly affect tail latency, so the median overhead stays near native. Because a 30-second window could hide a slow rise, we also repeated the 4 RPS runs with 300-second windows: the buffer fill levels off for both hidden states and queries/keys, with short bursts that return to a flat floor within seconds and no page waits.
The buffer design also matters when collection is under pressure. We ran MIA and DMI-Lib on the long-prompt workload of Appendix E.3, with each tool storing every request's captured data on local NVMe (DMI-Lib through its ClickHouse sink) and measured against the native vLLM. Under this workload, a page wait in MIA lasts until one page is freed, about 0.03 to 0.3 s, while DMI-Lib flushes its whole buffer before the step continues, about 1.4 to 2.3 s with 4 GiB and 3 to 5 s with 8 GiB. As a result, DMI-Lib falls behind from 12K-token prompts.
MIA: Median TPOT overhead over native vLLM, with average page waits per request Each cell: overhead (waits per request)
| Capture, buffer | 4K tokens | 8K | 12K | 16K |
|---|---|---|---|---|
| Hidden states, 2 GiB | +2.5 % (<0.1) | +2.8 % (0.5) | +8.8 % (1.6) | +8.9 % (2.7) |
| Hidden states, 4 GiB | +1.2 % (0) | +5.1 % (0) | +4.3 % (0.3) | +4.9 % (0.3) |
| Hidden states, 8 GiB | +3.7 % (0) | +4.5 % (0) | +2.0 % (0) | +5.0 % (0) |
| Queries/keys, 2 GiB | +4.8 % (0.1) | +25.9 % (1.4) | +68.1 % (2.4) | +195.7 % (3.6) |
| Queries/keys, 4 GiB | +5.9 % (0) | +6.4 % (0.2) | +13.7 % (0.8) | +17.7 % (2.2) |
| Queries/keys, 8 GiB | +4.4 % (0) | +5.2 % (0) | +7.4 % (<0.1) | +15.9 % (0.9) |
DMI-Lib: median TPOT overhead over native vLLM:
| Capture, buffer | 4K tokens | 8K | 12K | 16K |
|---|---|---|---|---|
| Hidden states, 2 GiB | +6.8 % | +19.0 % | saturated | saturated |
| Hidden states, 4 GiB | +8.0 % | +20.4 % | saturated | saturated |
| Hidden states, 8 GiB | +7.7 % | +19.1 % | saturated | saturated |
| Queries/keys, 2 GiB | +12.7 % | +762 % | saturated | saturated |
| Queries/keys, 4 GiB | +13.1 % | +157 % | saturated | saturated |
| Queries/keys, 8 GiB | +12.5 % | +23.4 % | saturated | saturated |
Waits increase as prompts lengthen and the buffer shrinks. TPOT rises once waits become frequent. With a 4 or 8 GiB buffer, hidden-state capture keeps TPOT within about 5 % of native at every tested prompt length. Capturing queries/keys writes more data per token but keeps TPOT within 8 % of native for prompts up to 12K tokens with sufficient buffer size. Because DMI-Lib drains its buffer to its back end more slowly, it saturates earlier.
Response to Q1 and Weakness 1: Mechanism and novelty beyond DMI-Lib and EasySteer
Our novelty lies in a combination of serving capabilities that none of the systems we compared offers: MIA works as a vLLM plugin that supports per-request configuration in a live server under CUDA graphs. Each request can independently configure capture (hidden states or queries/keys) or steering, along with the layers, tokens, and stages (prefill, decode or both) to which the operation applies. Requests with different configurations can share a batch and execute under CUDA graphs. To our knowledge, no existing system combines all of these together. vLLM-Hook and vLLM-Lens (v1.2.1, commit 7d252f7) offer per-request capture and steering only by giving up CUDA graphs. DMI-Lib (commit d03a09e) captures the hooks chosen before the run for every request. The runtime request selection in its Section 4.4 keeps or drops whole requests when its buffer is under pressure. EasySteer (commit 9c3c9d2) steers per request in its server under CUDA graphs. Its capture, however, runs only through its offline API, with one layer set per engine. It also needs a new capture graph whenever that set changes. Neither DMI-Lib nor EasySteer provides both capture and steering in one server. Moreover, MIA's plugin design requires no changes to the vLLM source code or model file to enable capture and steering.
The mechanism behind these capabilities is coordinating fine-grained capture/steer selection and reusable memory addresses without recapturing the graph. DMI-Lib and EasySteer use the same basic idea of small arrays at fixed device addresses that are rewritten between steps and read inside the graph. What MIA adds happens before each step: it resolves every request's selection, including work type, layers, token positions, prefill or decode against that step's batch layout and builds routing entries that map each selected row to a destination in a paged capture buffer. The replayed graph's capture operators read these entries and write only the selected rows into those pages in the buffer for flushing to CPU side. A background collector reassembles the rows by request and releases the pages for reuse. Steering requests use the same routing entries to select their vector and strength.
Capabilities in the versions we compared:
The remaining design choices mainly improve performance and could be replaced without losing any of these capabilities: fusing residual reconstruction with the capture write, uploading each step's routing entries in one transfer, etc. Appendix D shows some ablations to analyze the performance gain for these designs.