acceptodds
Under review as a conference paper at ICLR 2027

MIA: Runtime-Reconfigurable Access to Model Internals in LLM Inference Engines

Abstract

As Large Language Models (LLMs) and agents see broader deployment, runtime-time monitoring and intervention become increasingly important. This requires an inference engine to accept diverse requests asking for different layers, tokens, and vectors within one batched step, while meeting the latency targets. However, existing inference engines, which are primarily optimized for high-throughput text generation, are not designed to handle such heterogeneous request efficiently. Recent systems narrow this gap, but each pays for it by trading either throughput, write access, model support, or internal types. Therefore, we present MIA, a plugin for native vLLM server that captures and steers model internals at near-native efficiency. MIA consists of two primary parts, a Scheduler and a Courier. Before each step, the Scheduler decides what each request should read or write based on the batch and turns that decision into a routing entry. The GPU then replays the CUDA graph that reads and writes following those entries with minimal overhead. The Courier moves what the graph produced off the GPU in the background, returning it together with the model's own output. Under open-loop arrivals with requests asking for different layers, MIA generates each output token 2.9-14.1 faster than existing systems that support per-request hidden state capture during serving from 1 to 16 requests per second.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Inference engine, LLM, activation steering, model interpretability, memory management, vLLM, efficient inference system

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jF1vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents MIA, a plugin for vLLM that supports request-specific capture and steering of model internals while preserving full CUDA Graph replay. Its Aperture mechanism represents changing access requirements as routing entries consumed by operators installed before graph capture. A Scheduler maps request selections to the current batch and assigns capture destinations, while a Courier asynchronously offloads and reconstructs captured artifacts before releasing their storage. Experiments evaluate hidden-state capture, query/key capture, and activation steering under open-loop serving workloads, including heterogeneous layer selections and a four-GPU deployment of Llama-3.1-70B. The results show low median TPOT overhead and substantial improvements over callback-based access implementations.

Critical strengths

A coherent design for fine-grained access under optimized serving. Expressing request-specific selections and capture destinations as routing data provides a practical way to accommodate changing batches without inserting Python callbacks into graph replay. The design also explicitly addresses padded lanes, synchronization, request ownership, and delayed page reuse. These details matter because the usefulness of the interface depends on preserving both execution efficiency and correct data association.

Promising performance with meaningful experimental controls. MIA achieves low median TPOT overhead across capture and steering workloads and matches native vLLM’s SLO thresholds at the sampled arrival rates. Open-loop arrivals, controlled steering output lengths, idle-versus-active measurements, and the 70B tensor-parallel experiment strengthen the evidence that the implementation can be useful beyond a simple offline demonstration.

Critical weaknesses
  1. The technical contribution relative to the closest systems is not sufficiently isolated. DMI-Lib already combines graph-compatible capture operators, bounded GPU staging, asynchronous export, and runtime request selection. In particular, its Section 4.4 describes a batch-aligned keep/drop vector consumed by a GPU gather-compact operation without changing the captured graph. EasySteer v0.29.0 also documents heterogeneous per-request steering configurations and persistent in-graph payload and selection buffers. These capabilities do not establish equivalence to MIA’s heterogeneous capture-layer routing, but they narrow the novelty claim substantially. The strongest apparent distinction is coordinating fine-grained capture selections and reusable destinations without recapturing the graph. The paper should explain more precisely what additional mechanism makes this possible and distinguish its contribution from kernel fusion, batched transfers, and plugin integration. Existing microbenchmarks do not sufficiently isolate these contributions. This uncertainty is consequential because technical differentiation is central to my assessment. See DMI-Lib and the EasySteer v0.29.0 release description.

  2. The evaluation does not fully validate the system’s central access and collection guarantees. The heterogeneous experiment varies captured layers, while the steering benchmark uses identical vectors and strengths across requests. The paper does not report systematic checks of captured-value accuracy, request/token alignment under changing batches, or isolation between steered and unsteered requests. These are central requirements of the proposed interface. Furthermore, the 30-second measurement windows provide limited evidence about sustained operation when collection becomes the bottleneck acknowledged in Appendix B. Without buffer-occupancy, collection-wait, and artifact-completion measurements, it is difficult to determine whether low token-generation overhead also implies a stable and timely internal-data service. These gaps limit confidence in the broader serving claims.

Questions for authors
  1. Clarification of novelty: What specific mechanism is required beyond extending DMI-Lib’s runtime selection to finer granularity or EasySteer’s persistent selection buffers? Please distinguish capabilities unavailable in the evaluated baseline versions from implementation choices that primarily improve performance. Which baseline commits were used?

  2. Clarification and validation of access semantics: Does one engine support independently varying layer, token, and steering selections across requests, including concurrent capture and steering? What correctness checks establish numerical agreement, request/token alignment, and isolation under batch changes and repeated page reuse? Existing validation results would materially strengthen my assessment.

  3. Clarification of collection stability and latency accounting: Did the reported runs reach a stable capture-buffer occupancy and export rate? How are collection delays after the final generated token reflected in the reported metrics? Please provide available evidence on time waiting for free pages and the delay until a completed artifact becomes available.

5 comments
AuthorsOpenReview ↗

Response to Q1 and Weakness 1: Mechanism and novelty beyond DMI-Lib and EasySteer

Our novelty lies in a combination of serving capabilities that none of the systems we compared offers: MIA works as a vLLM plugin that supports per-request configuration in a live server under CUDA graphs. Each request can independently configure capture (hidden states or queries/keys) or steering, along with the layers, tokens, and stages (prefill, decode or both) to which the operation applies. Requests with different configurations can share a batch and execute under CUDA graphs. To our knowledge, no existing system combines all of these together. vLLM-Hook and vLLM-Lens (v1.2.1, commit 7d252f7) offer per-request capture and steering only by giving up CUDA graphs. DMI-Lib (commit d03a09e) captures the hooks chosen before the run for every request. The runtime request selection in its Section 4.4 keeps or drops whole requests when its buffer is under pressure. EasySteer (commit 9c3c9d2) steers per request in its server under CUDA graphs. Its capture, however, runs only through its offline API, with one layer set per engine. It also needs a new capture graph whenever that set changes. Neither DMI-Lib nor EasySteer provides both capture and steering in one server. Moreover, MIA's plugin design requires no changes to the vLLM source code or model file to enable capture and steering.

The mechanism behind these capabilities is coordinating fine-grained capture/steer selection and reusable memory addresses without recapturing the graph. DMI-Lib and EasySteer use the same basic idea of small arrays at fixed device addresses that are rewritten between steps and read inside the graph. What MIA adds happens before each step: it resolves every request's selection, including work type, layers, token positions, prefill or decode against that step's batch layout and builds routing entries that map each selected row to a destination in a paged capture buffer. The replayed graph's capture operators read these entries and write only the selected rows into those pages in the buffer for flushing to CPU side. A background collector reassembles the rows by request and releases the pages for reuse. Steering requests use the same routing entries to select their vector and strength.

Capabilities in the versions we compared:

MIADMI-Lib (d03a09e)EasySteer (9c3c9d2)
Per-request layers and tokens for capture, in a live server under CUDA graphsyesno, the same hooks for every requestno, offline capture with one layer set
Capture and steering in one serveryesno steeringits server only steers
Plugin on unmodified vLLMyesneeds its own model file copiesno, a fork

The remaining design choices mainly improve performance and could be replaced without losing any of these capabilities: fusing residual reconstruction with the capture write, uploading each step's routing entries in one transfer, etc. Appendix D shows some ablations to analyze the performance gain for these designs.

AuthorsOpenReview ↗

Response to Q2 and Weakness 2: Access semantics and correctness checks

Thank you for this question. Yes: in one MIA engine, each request independently chooses capture (hidden states or queries/keys) or steering, along with the layers, tokens and stages (prefill, decode or both) to which it applies, and for steering also its vector and strength. Capture and steering can also run concurrently in the same engine. Implementation details can viewed through the github repo in the paper.

Per reviewer's suggestion, our rebuttal adds a heterogeneous steering workload on Qwen2.5-1.5B-Instruct with ShareGPT prompts at 1 to 64 RPS. Each request draws its own setting: one of 16 vectors, a strength (2, 5 or 10), the layers (one, three or all 28), the tokens (all or only the last) and the phase (prefill, decode or both). One in five requests is not steered. Requests with different settings share each batch under CUDA graphs.

Median TPOT overhead over native vLLM 0.29 under heterogeneous steering (Qwen2.5-1.5B-Instruct):

1 RPS4 RPS8 RPS16 RPS32 RPS64 RPS
MIA+3.9 %+3.9 %+4.1 %+5.7 %+11.2 %+16.9 %
EasySteer+5.6 %+5.7 %+6.3 %+9.2 %+46.3 %saturated
Hotwire-vllm+6.1 %+7.7 %+10.2 %+28.7 %3.7×saturated
vLLM-Hook4.4×4.5×4.9×6.0×saturated–
vLLM-Lens8.4×43.1×saturated–––

Up to 16 RPS, heterogeneous steering costs MIA about as much as the uniform steering in Table 1 (+3.7 to 5.6 %). Beyond 16 RPS, the overhead rises to 11 % at 32 RPS and 17 % at 64 RPS. MIA still keeps up at 64 RPS. EasySteer and Hotwire-vllm also keep CUDA graphs. However, their overhead grows faster with load. Both fall behind at 64 RPS. vLLM-Hook and vLLM-Lens run in eager mode and reach 4.4× and 8.4× the native TPOT already at 1 RPS. All results and details are now in Appendix E.4.

Numerical Agreement

we compared MIA and native vLLM with Hugging Face Transformers (HF) in eager mode with bfloat16 precision under greedy decoding on ShareGPT prompts, capturing all 32 layers of Llama-3.1-8B and steering Qwen2.5-1.5B-Instruct.

Since HF and vLLM compute in bfloat16 with different kernels, native vLLM even differs from HF by 1.3 % in hidden states on a typical prompt and by up to 5.7 % on the worst one. This deviation sets the precision floor for any vLLM-based system. In eager mode, MIA's captured tensors and steered generations match with native vLLM's tensors. Under CUDA graphs, MIA's hidden states are remains close to native vLLM's hidden-state extraction. Its queries and keys also stay at the eager level. The tables below use HF in eager mode as the reference. Each cell reports the largest L2L_2 difference of native vLLM or MIA from HF. We did not collect queries/keys or evaluate steering for vLLM under CUDA graphs because it does not support these operations natively.

Largest relative L2L_2 difference of captured tensors from HF:

EagerCUDA graph
vLLMMIAvLLMMIA
Hidden states5.666 %5.666 %6.556 %6.558 %
Queries4.541 %4.541 %n/a5.028 %
Keys4.137 %4.137 %n/a4.783 %

Since steering is for token generation, we compare the log-probability each system assigns to each generated token before the outputs first diverge.

Largest log-probability difference of steered tokens from HF:

EagerCUDA graph
vLLMMIAvLLMMIA
Steering0.186 nats0.186 natsn/a0.198 nats

Request and Token Alignment

We served requests with heterogeneous capture settings at 8 RPS. Buffer pages were reused several times on average during this workload. We then decoded each captured last-layer hidden state into a token, since a row associated with the wrong request or token position could decode to a different token. The decoded token matches the emitted token for 99.6 % of vectors, compared with 99.7 % for native vLLM running each request alone in eager mode.

Isolation under Batch Change

We replayed every request of a heterogeneous steering run (16 RPS, prefix caching off) alone, once with its own steering and once with a different vector. Of the 142 requests whose replays can be told apart, 141 follow their own steering, and unsteered requests drift no more than steered ones, so we find no sign of steering leaking across requests.

Appendix C.5 has the full correctness checks, and Appendix E.4 the heterogeneous steering results and the isolation check.

AuthorsOpenReview ↗

Response to Q3 and Weakness 2 (1/2): Buffer stability under collection pressure

To address your concern, we traced the 4 GiB paged capture buffer at every engine step on the Table 1 workloads.

4 GiB buffer at 16 RPS (Llama-3.1-8B, all 32 layers):

Hidden statesQueries/keys
Mean buffer occupancy (%)3.4%61.1%
Median page waits per request (times)00.09
Median page wait latency (ms)0 ms6.4 ms
p99 page wait latency (ms)0 ms415 ms

Hidden-state capture never waits for a page from 1 to 16 RPS, so buffer pressure does not limit its near-native cost. Query/key capture stores more data per token than hidden-state capture and takes longer to write out of the buffer. At high load, it therefore runs close to the buffer's limit. These waits are brief for most requests and mainly affect tail latency, so the median overhead stays near native. Because a 30-second window could hide a slow rise, we also repeated the 4 RPS runs with 300-second windows: the buffer fill levels off for both hidden states and queries/keys, with short bursts that return to a flat floor within seconds and no page waits.

The buffer design also matters when collection is under pressure. We ran MIA and DMI-Lib on the long-prompt workload of Appendix E.3, with each tool storing every request's captured data on local NVMe (DMI-Lib through its ClickHouse sink) and measured against the native vLLM. Under this workload, a page wait in MIA lasts until one page is freed, about 0.03 to 0.3 s, while DMI-Lib flushes its whole buffer before the step continues, about 1.4 to 2.3 s with 4 GiB and 3 to 5 s with 8 GiB. As a result, DMI-Lib falls behind from 12K-token prompts.

MIA: Median TPOT overhead over native vLLM, with average page waits per request Each cell: overhead (waits per request)

Capture, buffer4K tokens8K12K16K
Hidden states, 2 GiB+2.5 % (<0.1)+2.8 % (0.5)+8.8 % (1.6)+8.9 % (2.7)
Hidden states, 4 GiB+1.2 % (0)+5.1 % (0)+4.3 % (0.3)+4.9 % (0.3)
Hidden states, 8 GiB+3.7 % (0)+4.5 % (0)+2.0 % (0)+5.0 % (0)
Queries/keys, 2 GiB+4.8 % (0.1)+25.9 % (1.4)+68.1 % (2.4)+195.7 % (3.6)
Queries/keys, 4 GiB+5.9 % (0)+6.4 % (0.2)+13.7 % (0.8)+17.7 % (2.2)
Queries/keys, 8 GiB+4.4 % (0)+5.2 % (0)+7.4 % (<0.1)+15.9 % (0.9)

DMI-Lib: median TPOT overhead over native vLLM:

Capture, buffer4K tokens8K12K16K
Hidden states, 2 GiB+6.8 %+19.0 %saturatedsaturated
Hidden states, 4 GiB+8.0 %+20.4 %saturatedsaturated
Hidden states, 8 GiB+7.7 %+19.1 %saturatedsaturated
Queries/keys, 2 GiB+12.7 %+762 %saturatedsaturated
Queries/keys, 4 GiB+13.1 %+157 %saturatedsaturated
Queries/keys, 8 GiB+12.5 %+23.4 %saturatedsaturated

Waits increase as prompts lengthen and the buffer shrinks. TPOT rises once waits become frequent. With a 4 or 8 GiB buffer, hidden-state capture keeps TPOT within about 5 % of native at every tested prompt length. Capturing queries/keys writes more data per token but keeps TPOT within 8 % of native for prompts up to 12K tokens with sufficient buffer size. Because DMI-Lib drains its buffer to its back end more slowly, it saturates earlier.

Reviewer_A8HtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MIA is a vLLM plugin that adds per-request capture of hidden states and Q/K vectors, plus activation steering, while keeping full CUDA graph replay. It works in three parts:

  • Routing buffer: each request's choices are written as data at fixed addresses, and operators recorded in the graph read them.
  • Scheduler: maps each step's requests to slots in a bounded paged buffer.
  • Courier: offloads captured rows to the host asynchronously and returns them per request, via RPC or an NVMe file.

On H100s with ShareGPT traffic, median TPOT overhead is 4.9–6.4% for HS capture, 6.7–8.4% for QK capture, and 3.7–5.6% for steering, and native vLLM's SLO knee is preserved. Under heterogeneous requests, MIA is 2.9–19.6× faster than eager-mode systems (vLLM-Lens, vLLM-Hook).

Critical strengths
  1. Practical gap, clean design. MIA is the only system in Table 2 that supports per-request capture and steering with full graph replay on an unforked engine, at low overhead.
  2. Broad evaluation. It compares against ~8 systems and includes an overhead decomposition, kernel microbenchmarks, and a 70B TP=4 run.
Critical weaknesses
  1. Incremental novelty and incomplete baselines. EasySteer already uses graph-safe device tables, and DMI-Lib already captures in-graph with asynchronous offload.
    • EasySteer is missing from all capture experiments.
    • The headline heterogeneous speedup is measured only against eager-mode systems, so it mostly reflects eager vs. graph execution.
    • Baselines run on different vLLM versions (0.11/0.21/0.23) but are all normalized to 0.29, which confounds the DMI-Lib comparison.
  2. The evaluation does not match the motivation.
    • Artifacts arrive only after a request finishes, so online monitor → steer is not supported.
    • Long contexts and buffer backpressure, the stated main limitation, are not stress-tested.
    • The correctness of captured or steered values against a reference is never verified.

Minor: reported numbers are inconsistent (HS 4.9–8.4% vs. 4.9–6.4%; 100 vs. 121 MiB), and there are typos ("runtimetime", "senario").

Questions for authors
  1. Are captured HS/QK values and steered outputs numerically consistent with a reference implementation?
  2. Why is EasySteer excluded from the capture experiments? Can DMI-Lib be normalized against its own vLLM 0.11?
  3. How does MIA perform with long contexts (8K+), different buffer sizes, and page-stall rates, and what is the delay before artifacts are available?
  4. Can captured values be streamed during generation to support closed-loop monitoring?
5 comments
AuthorsOpenReview ↗

Response to Q1 and Weakness 2: Numerical consistency with a reference

Following your suggestion, we compared captured steered data from MIA and native vLLM with Hugging Face Transformers (HF) in eager mode with bfloat16 precision under greedy decoding on ShareGPT prompts, capturing all 32 layers of Llama-3.1-8B and steering Qwen2.5-1.5B-Instruct.

Since HF and vLLM compute in bfloat16 with different kernels, native vLLM even differs from HF by 1.3 % in hidden states on a typical prompt and by up to 5.7 % on the worst one. This deviation sets the precision floor for any vLLM-based system. In eager mode, MIA's captured tensors and steered generations match with native vLLM's tensors. Under CUDA graphs, MIA's hidden states are remains close to native vLLM's hidden-state extraction. Its queries and keys also stay at the eager level. The tables below use HF in eager mode as the reference. Each cell reports the largest L2L_2 difference of native vLLM or MIA from HF. We did not collect queries/keys or evaluate steering for vLLM under CUDA graphs because it does not support these operations natively.

Largest relative L2L_2 difference of captured tensors from HF:

EagerCUDA graph
vLLMMIAvLLMMIA
Hidden states5.666 %5.666 %6.556 %6.558 %
Queries4.541 %4.541 %n/a5.028 %
Keys4.137 %4.137 %n/a4.783 %

Since steering is for token generation, we compare the log-probability each system assigns to each generated token before the outputs first diverge.

Largest log-probability difference of steered tokens from HF:

EagerCUDA graph
vLLMMIAvLLMMIA
Steering0.186 nats0.186 natsn/a0.198 nats

To address concerns that shared pages and asynchronous copies may cause errors, we also served requests with heterogeneous per-request capture settings at 8 requests per second. The captured last-layer hidden states predict the emitted token for 99.6 % of vectors, compared with 99.7 % for an eager reference. This agreement supports the association of captured data with the correct request and token. All results and details are now in Appendix C.5.

AuthorsOpenReview ↗

Response to Q2 and Weakness 1: EasySteer in the capture experiments and DMI-Lib's engine version

Regarding your first question, we apologize for the unclear and misleading description of EasySteer in Section 2.2 and Table 2. Section 2.2 stated that EasySteer supports graph-compatible capture and steering through persistent device tables. Table 2 marked it as supporting both a server and graph-safe capture, suggesting that capture was available through its server. However, based on our understanding of EasySteer's current documentation and code, EasySteer uses its offline Python API for capture. Its server accepts per-request steering but offers no capture interface. Capture uses the same layer set for every request in the engine and requires rebuilding the CUDA graph whenever that layer set changes. EasySteer also does not capture queries/keys. We have corrected Section 2.2 and Table 2. We exclude EasySteer from the serving capture experiments because it offers no server capture interface.

For your second question, we are sorry for a label typo in the submitted Table 1: it listed DMI-Lib's engine as a fork of vLLM 0.11. DMI-Lib is in fact built on a fork of vLLM 0.17.1. To separate MIA's design from the engine version, we built MIA on vLLM 0.17.1 and compared the two on the same engine. In the table, MIA's overhead is at or below DMI-Lib's at every rate. The difference widens with load.

Median TPOT overhead over native vLLM 0.29 (Llama-3.1-8B, all 32 layers, every token):

1 RPS4 RPS8 RPS16 RPS
Hidden states, DMI-Lib (fork of 0.17.1)+6.2 %+6.8 %+7.1 %+14.2 %
Hidden states, MIA (vLLM 0.17.1)+5.1 %+6.3 %+7.1 %+7.2 %
Queries/keys, DMI-Lib (fork of 0.17.1)+8.2 %+9.7 %+10.3 %+16.8 %
Queries/keys, MIA (vLLM 0.17.1)+6.9 %+8.5 %+9.1 %+9.3 %

The revised Table 1 corrects DMI-Lib's label. The new Appendix C.4 compares MIA on vLLM 0.17.1 with DMI-Lib.

AuthorsOpenReview ↗

Response to Q3 and Weakness 2: Long contexts, buffer sizes, page stalls and artifact delay

Thank you for raising the concern. We added two experiments to address it.

For long contexts and buffer sizes, we ran MIA and DMI-Lib on the long-prompt workload of Appendix E.3, with each tool storing every request's captured data on local NVMe (DMI-Lib through its ClickHouse sink) and measured against the native vLLM. Under this workload, a page wait in MIA lasts until one page is freed, about 0.03 to 0.3 s, while DMI-Lib flushes its whole buffer before the step continues, about 1.4 to 2.3 s with 4 GiB and 3 to 5 s with 8 GiB. As a result, DMI-Lib falls behind from 12K-token prompts.

MIA: Median TPOT overhead over native vLLM, with average page waits per request Each cell: overhead (waits per request)

Capture, buffer4K tokens8K12K16K
Hidden states, 2 GiB+2.5 % (<0.1)+2.8 % (0.5)+8.8 % (1.6)+8.9 % (2.7)
Hidden states, 4 GiB+1.2 % (0)+5.1 % (0)+4.3 % (0.3)+4.9 % (0.3)
Hidden states, 8 GiB+3.7 % (0)+4.5 % (0)+2.0 % (0)+5.0 % (0)
Queries/keys, 2 GiB+4.8 % (0.1)+25.9 % (1.4)+68.1 % (2.4)+195.7 % (3.6)
Queries/keys, 4 GiB+5.9 % (0)+6.4 % (0.2)+13.7 % (0.8)+17.7 % (2.2)
Queries/keys, 8 GiB+4.4 % (0)+5.2 % (0)+7.4 % (<0.1)+15.9 % (0.9)

DMI-Lib: median TPOT overhead over native vLLM:

Capture, buffer4K tokens8K12K16K
Hidden states, 2 GiB+6.8 %+19.0 %saturatedsaturated
Hidden states, 4 GiB+8.0 %+20.4 %saturatedsaturated
Hidden states, 8 GiB+7.7 %+19.1 %saturatedsaturated
Queries/keys, 2 GiB+12.7 %+762 %saturatedsaturated
Queries/keys, 4 GiB+13.1 %+157 %saturatedsaturated
Queries/keys, 8 GiB+12.5 %+23.4 %saturatedsaturated

Waits increase as prompts lengthen and the buffer shrinks. TPOT rises once waits become frequent. With a 4 or 8 GiB buffer, hidden-state capture keeps TPOT within about 5 % of native at every tested prompt length. Capturing queries/keys writes more data per token but keeps TPOT within 8 % of native for prompts up to 12K tokens with sufficient buffer size. Because DMI-Lib drains its buffer to its back end more slowly, it saturates earlier.

For the delay, we measured the time from a request's last token until all of its captured data are assembled and stored on the server, on ShareGPT prompts with all 32 layers captured at every token.

Median time from the last token until captured hidden states are stored (Llama-3.1-8B):

1 RPS4 RPS8 RPS16 RPS
MIA0.20 s0.24 s0.30 ssaturated
DMI-Lib (stores to ClickHouse)0.24 s4.85 ssaturatedsaturated

We further model what limits this delay in Appendix C.6. The GPU produces captured data at RGPU=λreq⋅VreqR_{\text{GPU}} = \lambda_{\text{req}} \cdot V_{\text{req}}, where λreq\lambda_{\text{req}} is the request rate and VreqV_{\text{req}} the captured volume per request (121 MiB for hidden states on Llama-3.1-8B). The CPU side collects it at

RCPU=min⁡(Blink, VreqVwrite/Bwrite+Vread/Bread, Nworker⋅Rworker)R_{\text{CPU}} = \min\left(B_{\text{link}},\ \frac{V_{\text{req}}}{V_{\text{write}}/B_{\text{write}} + V_{\text{read}}/B_{\text{read}}},\ N_{\text{worker}} \cdot R_{\text{worker}}\right)

the slowest of three factors. The first is the GPU-to-host bandwidth (BlinkB_{\text{link}}). The second term is the data rate that storage can absorb, where one request writes VwriteV_{\text{write}} bytes and reads VreadV_{\text{read}} bytes at the drive's write and read bandwidths BwriteB_{\text{write}} and BreadB_{\text{read}} for processing per-request captured data that will be sent back to clients. The last term is the throughput of the background collection workers (NworkerN_{\text{worker}} of them), each processing RworkerR_{\text{worker}} bytes per second. Usually, the storage is the tighter limit. To show what this limit means in practice, the table below gives the highest request rate that each single drive's peak bandwidth allows, for the drive type in our server and two other ones.

Highest rate of 121 MiB hidden-state requests that each drive sustains from vendor peak bandwidth:

StorageRead / write (GB/s)Highest rate
Samsung PM1733a, PCIe Gen47.5 / 4.113 RPS
Samsung PM1743, PCIe Gen514.0 / 6.019 RPS
Micron 9550 PRO, PCIe Gen514.0 / 10.029 RPS

The full model and measurements are in Appendix C.6, and the long-context results, including the DMI-Lib comparison, in Appendix E.3.

Reviewer_3F6EOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents MIA, a vLLM plugin that lets each request in a batch read (capture) or modify (steer) the model's internal activations while the engine keeps its fast CUDA-graph execution.

  1. Architecture: the Aperture installs capture and steering operators before graph capture, and during replay they read routing entries that the Scheduler prepares for each step. Captured data goes into a bounded paged buffer, and the Courier, a background thread, moves it to host memory and returns it per request.

  2. Claimed contributions: the Aperture routing interface, which makes per-request access compatible with graph replay; the Scheduler and Courier, which manage it under continuous batching; and an implementation on unmodified vLLM.

  3. Validation: the evaluation compares capture on the 8B model and steering on the 1.5B model with prior systems, then runs all three workloads on a 70B model against its own native vLLM baseline. At the rates listed in the main results table, MIA adds single-digit median TPOT overhead on these 8B capture and 1.5B steering workloads and matches native vLLM's serving capacity. Under mixed per-request layer selections, hidden-state capture with MIA is much faster than with the eager-mode baselines.

Critical strengths
  1. The method keeps vLLM's full CUDA graph replay while letting each request choose what to capture or steer, at close to native serving cost. The operators and their buffer addresses stay fixed after graph capture, while the Scheduler updates routing entries before each step. The paper reports that this keeps the same serving capacity as native vLLM for both capture and steering. In the paper's capability comparison, no other system offers graph-safe capture and steering on an unmodified engine. This matters because it makes runtime monitoring and steering practical inside a production serving engine.

  2. The overhead analysis is careful enough to show where the cost comes from, not only how large it is. On the hidden-state capture workload, the comparison of idle and active runs attributes most of the measured overhead to the installed operators rather than to moving captured data. It also measures the fused capture and steering kernels against equivalent PyTorch operations, and repeats the three workloads on a 70B model under four-way tensor parallelism. This matters because it shows which part of the design carries the cost and supports the paper's claim of low overhead at the reported rates.

Critical weaknesses
  1. The paper claims accurate access to model internals but reports no quantitative check of the captured values or steered outputs against a reference. The introduction states that the Scheduler and Courier guarantee accurate and efficient model internal operations. The evaluation measures latency, serving capacity and kernel time. I found no comparison of MIA's captures with an eager-mode reference, or of its steered generations with a reference steering implementation. The design fuses residual reconstruction with capture, shares a page index across layers and copies captured data asynchronously. These mechanisms make numerical and request-ownership validation important. This matters because a probe or monitor built on these outputs inherits any capture error, so I cannot yet trust the accuracy claim.

  2. The comparison with DMI-Lib, the closest graph-preserving capture baseline, leaves the engine version's share of the latency gap unresolved. Against DMI-Lib, which also keeps CUDA graph replay but has no per-request selection, both systems match the capture SLO knee, and the paper explains MIA's latency advantage through direct paged-buffer writes and fused residual reconstruction. DMI-Lib runs on a fork of vLLM 0.11, while MIA runs on 0.29. The reported engine control compares 0.21 with 0.29. I found no corresponding control for DMI-Lib's 0.11 engine, so its contribution to the gap remains unresolved. This matters because this comparison, not the speedup over eager-mode tools, is the evidence that MIA's design improves on prior graph-preserving capture.

  3. The evaluation leaves two parts of the runtime-reconfiguration claim untested: heterogeneous steering and token selections, and long requests under buffer pressure. The mixed-request experiment varies only the number of captured hidden-state layers, the steering workload applies one unit-norm vector at every layer, and I found no reported run that varies phase or token selections. All serving runs draw short chat requests from a fixed pool and use the default 4 GiB capture buffer, while the paper acknowledges that a full buffer stalls the engine. Full hidden-state capture of an average 8B request already takes about 121 MiB, and the paper motivates the design with an 8192-token request on the 70B model that needs 10 GiB. This matters because I cannot tell whether the near-native cost holds for the heterogeneous steering and long-request monitoring that motivate the paper.

Questions for authors
  1. Did you compare MIA's captured hidden states and queries and keys with those of an eager-mode reference on the same inputs, and steered generations with a reference steering implementation under greedy decoding? If so, what were the largest differences? A match within bfloat16 precision would resolve Weakness 1 and raise my soundness score.

  2. Does any existing run compare DMI-Lib with MIA on the same vLLM version, or measure native vLLM 0.11 against 0.29 under the same serving settings? Either would let me assess how much of the margin comes from MIA's design rather than the engine, and would resolve Weakness 2. A run on a closer but still different version would reduce this concern but not resolve it.

  3. How often did the engine wait for free capture pages in the reported runs, and how full did the 4 GiB buffer get at the highest reported load? Low occupancy and negligible page-wait time would show that buffer pressure does not limit the reported workload, while substantial wait time would weaken the near-native-cost result at those rates. Neither answer would settle longer requests, so that part of Weakness 3 would remain.

  4. Did any reported run use different steering vectors or strengths across requests, or different phase and token selections? The answer would tell me how broadly to read the mixed-request results and how much of Weakness 3 applies.

4 comments
AuthorsOpenReview ↗

Response to Q1 and Weakness 1: Correctness of captured values and steered outputs

Following your suggestion, we compared MIA and native vLLM with Hugging Face Transformers (HF) in eager mode with bfloat16 precision under greedy decoding on ShareGPT prompts, capturing all 32 layers of Llama-3.1-8B and steering Qwen2.5-1.5B-Instruct.

Since HF and vLLM compute in bfloat16 with different kernels, native vLLM even differs from HF by 1.3 % in hidden states on a typical prompt and by up to 5.7 % on the worst one. This deviation sets the precision floor for any vLLM-based system. In eager mode, MIA's captured tensors and steered generations match with native vLLM's tensors. Under CUDA graphs, MIA's hidden states are remains close to native vLLM's hidden-state extraction. Its queries and keys also stay at the eager level. The tables below use HF in eager mode as the reference. Each cell reports the largest L2L_2 difference of native vLLM or MIA from HF. We did not collect queries/keys or evaluate steering for vLLM under CUDA graphs because it does not support these operations natively.

Largest relative L2L_2 difference of captured tensors from HF:

EagerCUDA graph
vLLMMIAvLLMMIA
Hidden states5.666 %5.666 %6.556 %6.558 %
Queries4.541 %4.541 %n/a5.028 %
Keys4.137 %4.137 %n/a4.783 %

Since steering is for token generation, we compare the log-probability each system assigns to each generated token before the outputs first diverge.

Largest log-probability difference of steered tokens from HF:

EagerCUDA graph
vLLMMIAvLLMMIA
Steering0.186 nats0.186 natsn/a0.198 nats

To address concerns that shared pages and asynchronous copies may cause errors, we also served requests with heterogeneous per-request capture settings at 8 requests per second. The captured last-layer hidden states predict the emitted token for 99.6 % of vectors, compared with 99.7 % for an eager reference. This agreement supports the association of captured data with the correct request and token. All results and details are now in Appendix C.5.

AuthorsOpenReview ↗

Response to Q2 and Weakness 2: Engine version in the DMI-Lib comparison

We apologize for an versiorn typo in Table 1 of the submitted paper, which listed DMI-Lib's engine as a fork of vLLM 0.11 and may have caused confusion about the comparison. DMI-Lib is built on a fork of vLLM 0.17.1. We have corrected this error in the revised Table 1. To address your concern that the performance gain could come from the vLLM engine version, we built MIA on vLLM v0.17.1 and compared it with DMI-Lib. With both systems using the same engine version, MIA's overhead is at or below DMI-Lib's at every tested request rate, and the gap widens as load increases.

Median TPOT overhead over native vLLM 0.29 (Llama-3.1-8B, all 32 layers, every token):

1 RPS4 RPS8 RPS16 RPS
Hidden states, DMI-Lib (fork of v0.17.1)+6.2 %+6.8 %+7.1 %+14.2 %
Hidden states, MIA (vLLM v0.17.1)+5.1 %+6.3 %+7.1 %+7.2 %
Queries/keys, DMI-Lib (fork of v0.17.1)+8.2 %+9.7 %+10.3 %+16.8 %
Queries/keys, MIA (vLLM v0.17.1)+6.9 %+8.5 %+9.1 %+9.3 %

The revised Table 1 corrects DMI-Lib's label. The new Appendix C.4 compares MIA on vLLM 0.17.1 with DMI-Lib.

AuthorsOpenReview ↗

Response to Q3 and Weakness 3: Buffer pressure and long requests

Thank you for this question. The submitted paper left this gap open. We traced the 4 GiB paged capture buffer at every engine step on the Table 1 workloads.

4 GiB buffer at 16 RPS (Llama-3.1-8B, all 32 layers):

Hidden statesQueries/keys
Mean buffer occupancy (%)3.4%61.1%
Median page waits per request (times)00.09
Median page wait latency (ms)0 ms6.4 ms
p99 page wait latency (ms)0 ms415 ms

Hidden-state capture never waits for a page from 1 to 16 RPS, so buffer pressure does not limit its near-native cost. Query/key capture stores more data per token than hidden-state capture and takes longer to write out of the buffer. At high load, it therefore runs close to the buffer's limit. These waits are brief for most requests and mainly affect tail latency, so the median overhead stays near native.

The buffer design also matters when collection is under pressure. We ran MIA and DMI-Lib on the long-prompt workload of Appendix E.3, with each tool storing every request's captured data on local NVMe (DMI-Lib through its ClickHouse sink) and measured against the native vLLM. Under this workload, a page wait in MIA lasts until one page is freed, about 0.03 to 0.3 s, while DMI-Lib flushes its whole buffer before the step continues, about 1.4 to 2.3 s with 4 GiB and 3 to 5 s with 8 GiB. As a result, DMI-Lib falls behind from 12K-token prompts.

Median TPOT overhead over native vLLM, with average page waits per request Each cell: overhead (waits per request)

Capture, buffer4K tokens8K12K16K
Hidden states, 2 GiB+2.5 % (<0.1)+2.8 % (0.5)+8.8 % (1.6)+8.9 % (2.7)
Hidden states, 4 GiB+1.2 % (0)+5.1 % (0)+4.3 % (0.3)+4.9 % (0.3)
Hidden states, 8 GiB+3.7 % (0)+4.5 % (0)+2.0 % (0)+5.0 % (0)
Queries/keys, 2 GiB+4.8 % (0.1)+25.9 % (1.4)+68.1 % (2.4)+195.7 % (3.6)
Queries/keys, 4 GiB+5.9 % (0)+6.4 % (0.2)+13.7 % (0.8)+17.7 % (2.2)
Queries/keys, 8 GiB+4.4 % (0)+5.2 % (0)+7.4 % (<0.1)+15.9 % (0.9)

DMI-Lib: median TPOT overhead over native vLLM:

Capture, buffer4K tokens8K12K16K
Hidden states, 2 GiB+6.8 %+19.0 %saturatedsaturated
Hidden states, 4 GiB+8.0 %+20.4 %saturatedsaturated
Hidden states, 8 GiB+7.7 %+19.1 %saturatedsaturated
Queries/keys, 2 GiB+12.7 %+762 %saturatedsaturated
Queries/keys, 4 GiB+13.1 %+157 %saturatedsaturated
Queries/keys, 8 GiB+12.5 %+23.4 %saturatedsaturated

Waits increase as prompts lengthen and the buffer shrinks. TPOT rises once waits become frequent. With a 4 or 8 GiB buffer, hidden-state capture keeps TPOT within about 5 % of native at every tested prompt length. Capturing queries/keys writes more data per token but keeps TPOT within 8 % of native for prompts up to 12K tokens with sufficient buffer size. Because DMI-Lib drains its buffer to its back end more slowly, it saturates earlier. The long-request results, including the DMI-Lib comparison, are now in Appendix E.3.

AuthorsOpenReview ↗

Response to Q4 and Weakness 3: Heterogeneous steering

Thank you for raising your concern. Throughout the initial submission, on steering tasks, we have used the same vector and strength; however, MIA is indeed capable of handling heterogeneous steering requests as well. Following your advice, we built a heterogeneous steering workload on Qwen2.5-1.5B-Instruct with ShareGPT prompts at 1 to 64 RPS. Each request draws its own setting: one of 16 steering vectors, a strength (2, 5 or 10), the layers (one, three or all 28), the tokens (all or only the last), and the stage (prefill, decode, or both). One in five requests is not steered. Requests with different settings share each batch under CUDA graphs.

Median TPOT overhead over native vLLM 0.29 under heterogeneous steering (Qwen2.5-1.5B-Instruct):

1 RPS4 RPS8 RPS16 RPS32 RPS64 RPS
MIA+3.9 %+3.9 %+4.1 %+5.7 %+11.2 %+16.9 %
EasySteer+5.6 %+5.7 %+6.3 %+9.2 %+46.3 %saturated
Hotwire-vllm+6.1 %+7.7 %+10.2 %+28.7 %3.7×saturated
vLLM-Hook4.4×4.5×4.9×6.0×saturated–
vLLM-Lens8.4×43.1×saturated–––

Up to 16 RPS, heterogeneous steering costs MIA about as much as the uniform steering in Table 1 (+3.7 to 5.6 %). Beyond 16 RPS, the overhead rises to 11 % at 32 RPS and 17 % at 64 RPS. MIA still keeps up at 64 RPS. EasySteer and Hotwire-vllm also keep CUDA graphs. However, their overhead grows faster with load. Both fall behind at 64 RPS. vLLM-Hook and vLLM-Lens run in eager mode and reach 4.4× and 8.4× the native TPOT already at 1 RPS. All results and details are now in Appendix E.4.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Follow-up on Author Responses

Dear Reviewers,

As the discussion deadline is approaching, we want to send a reminder that we have addressed all reviewer comments and would appreciate any further thoughts or follow-up questions. We are happy to provide additional clarification and continue the discussion.

Thank you again for your time and consideration.

Thank you for the detailed rebuttal and the additional experiments. The numerical-consistency results, the corrected EasySteer/DMI-Lib comparison on the same engine, and the long-context study resolve most of my concerns about soundness.

My remaining concern, and the one that most affects my rating, is motivation: why capture needs to happen inside the serving engine at all.

Since MIA does not support streaming or closed-loop monitoring, the captured activations are consumed only after a request finishes. For a decoder-only model, a single teacher-forced forward pass over prompt + response reproduces those activations, up to the same bf16-level numerical differences your rebuttal reports. That pass is a prefill, which is compute-bound, highly parallel, and far cheaper than autoregressive decoding. An offline tool such as HF, TransformerLens, or NNsight could therefore produce the same artifacts, with richer access and no change to the serving path. For capture specifically, it is unclear what in-engine capture buys over "generate with native vLLM, then re-run a prefill pass."

To address this, I would like to see:

  1. A two-pass baseline. Native vLLM generates; a second prefill-only pass over prompt + response extracts the same layers, using vLLM's native HS extraction or HF. Please compare total GPU time, throughput, and end-to-end latency to artifact availability against MIA on the same workload.
  2. A concrete use case where in-engine capture is necessary, i.e., one where post-hoc recomputation is infeasible or substantially more expensive. Steering clearly needs to be in-engine, but that is already covered by EasySteer and Hotwire. If the main justification for capture is cost savings at production scale, please quantify it.

If the authors can convincingly prove the motivation, I am willing to raise my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.