- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents AsyncLLM, a training-free framework for concurrent LLM inference with shared memory. The extensions to hybrid and multimodal models are technically useful, and the applications demonstrate feasibility. However, the experimental evidence does not sufficiently isolate the method’s benefits.
Critical strengths
-
Useful technical contribution. The composable cache framework extends shared-memory concurrent inference to hybrid and multimodal models through Gated DeltaNet state composition and multimodal positional handling, exposing these capabilities through a unified programming interface.
-
Broad applicability without task-specific training. Demonstrations spanning asynchronous text or image updates, streaming video, videogames, and system monitoring provide meaningful evidence that existing models can support diverse asynchronous workflows within the same framework.
Critical weaknesses
-
The contribution of shared memory is not isolated. Table 1 compares Qwen-based AsyncLLM with Mage-VL, changing both the underlying model and inference framework. The same-model comparisons in games and monitoring compare different agent designs without isolating shared memory. A matched asynchronous baseline using ordinary text communication, with the same model and coroutine structure, is needed to establish what specifically benefits from the proposed mechanism.
-
The responsiveness claims need direct latency measurements. Figure 4 and Table 2 measure delay using forward-pass counts, treating multiple tokens generated in one batch as one pass. However, pass duration varies with concurrency. The video throughput results establish processing capacity, but do not establish observation-to-action latency in games or monitoring. Please report wall-clock reward or accuracy versus response-latency curves on fixed hardware, including total inference cost.
Questions for authors
Please check the Weaknesses section.
6 comments
Text-based communication and latency (follow-up 1/2)
As we promised earlier, we implemented text-based communication in several environments to benchmark effectiveness and latency.
Text-based Asynchronous Reasoning
To better decouple different moving parts, we begin with text-based communication in the setup from Section 4.1 with k=0, i.e. solving a fully defined problem from MATH-500 on H200 with Qwen3.5-27B. To run the extra compute-heavy experiments on time, we test this section using 100 random samples out of MATH-500 since the differences we observed are large enough. We start with the same asynchronous reasoning setup where the ‘writer’ can write partial responses while the ‘thinker’ is still reasoning about the problem, thus solving the problem faster.
We first attempted several straightforward ways to support text-based communication: letting two agents message each other using tools or running our standard memory pipeline with re-encoding new text tokens that are accessible to the thinker/writer coroutines without sharing memory states. However, we found that both versions introduced significant real-time delays where every minor update to the memory state required processing tens of extra tokens.
To alleviate this, we implemented a modified protocol where the thinker coroutine proceeded normally, but the writer switched only pulled updates every 50 steps and ran standard inference between them. This improved baseline achieved significantly smaller performance drawdowns without compromising accuracy, though it is still significantly slower than shared cache.
Table C: Asynchronous Reasoning, 100 samples from MATH-500, Qwen3.5-27B, H200
| Setup | Accuracy | Mean latency |
|---|---|---|
| AsyncLLM (main, shared cache) | 94% | 72.98 s |
| Text-based communication as-is | 95% | 6194.96 s |
| Text-based every 50 writer steps | 94% | 257.72 s |
To summarize, asynchronous agents with text-based communication obtain similar accuracy scores, but take significantly longer per sample due to the additional computations for passing and re-encoding tokens. Straightforward text-based conversion resulted in very long delays (~1.5 hours) that could be reduced by redesigning the agent to communicate less often.
Text-based Streaming Video Understanding
For streaming video understanding, naive token-based communication without changing the agent architecture likewise ran several times slower on test problems and we would be unable to evaluate it on the entire dataset within our compute budget. However, we got significantly better results by redesigning the text-based agent using AsyncLLM that only passes messages when the probe found a description-worthy event.
Table D: improved text-based agent on ProactiveVideoQA, Qwen3-9B, 1x H200.
| Setup | WEB | EGO | TV | VAD | Frame latency | Wall time |
|---|---|---|---|---|---|---|
| AsyncLLM (main, shared cache) | 49.3 | 56.3 | 63.8 | 35.8 | 0.78s | 6.5s |
| AsyncLLM (text-based, optimized) | 47.2 | 56.7 | 60.5 | 32.34 | 1.23s | 7.6s |
In Table D above, we report accuracy on ProactiveVideoQA subsets and two latency metrics. Frame latency is how long it takes to process a single video frame, from receiving the image to deciding whether or not it has a new event to describe, including the time it takes to communicate the result. Wall time is the average time to process a StreamingVideoQA sample.
Overall, the optimized text-based agent has comparable accuracy scores across subsets, but takes ~58% longer to reason about a given frame and ~17% slower overall. Note that wall time is less affected by video delays because text-based AsyncLLM can still schedule multiple parallel frames into the same batch, with denser workload compensating for delays.
Text-based VizDoom
Similarly, we only evaluate the optimized agent where the short-term action coroutine only re-encodes context after taking its action, which significantly reduces GPU latency but introduces slight thought-action delays.
Table A2: DoomHealthGathering latency, Qwen-3.6-35B-A3B, H200
| Method | Reward | Forward / action | Seconds / action |
|---|---|---|---|
| Probe | 508.9 | 1 | 0.06 |
| Sequential | 575.81 | 132.76 | 4.59 |
| AsyncLLM (main) | 573.45 | 17.26 | 0.58 |
| AsyncLLM (self-defined) | 589.78 | 41.90 | 1.46 |
| AsyncLLM (text-based) | 573.81 | 16.95 | 2.20 |
Table B2: DoomDeadlyCorridor latency, Qwen-3.6-35B-A3B, H200
| Method | Reward | Forward / action | Seconds / action |
|---|---|---|---|
| Probe | 184.49 | 1 | 0.07 |
| Sequential | 340.10 | 116.88 | 4.25 |
| AsyncLLM (main) | 362.24 | 24.57 | 0.96 |
| AsyncLLM (self-defined) | 297.88 | 49.66 | 2.04 |
| AsyncLLM (text-based) | 354.8 | 19.28 | 3.79 |
We evaluate these text-based agents using the same setups as in our previous response and report them in tables A2, B2, at the bottom row. Both text-based agents got reasonable scores, but they had to play a much slower version of the game to match with their decision-making speed. Additionally, the thought-action delay from optimized reencoding appears to have had some effect on the ‘shooter’ scenario in DoomDeadlyCorridor.
DevOpsGym Monitoring (follow-up 2/2)
Your review requested additional latency benchmarks for Figure 4 (VizDoom) and Table 2 (Monitoring). Earlier, we reported VizDoom latency in our initial response (Tables A, B) and promised to re-run our monitoring evaluation on the same GPU type so you can compare real-world latency, and we report these results now. We measure the average total time it takes to process a full scenario if the events are fed as soon as the agent is ready to ingest them.
Table E: DevOpsGym Monitoring, Qwen3.6-35B-A3B Monitoring, 1 H200
| Setup | Accuracy (%) | Wall time, s. |
|---|---|---|
| Q3.6-35B-A3B | 61.76 | 204.47 |
| + early answer | 47.06 | 162.58 |
| + skip rows | 26.47 | 90.58 |
| AsyncLLM | 55.88 | 97.56 |
In the table above, the relative speed-up is slightly (<10%) worse in latency than it was in terms of forward passes due to slight parallelism overhead, but AsyncLLM still maintains a strong advantage over sequential baselines. Only the skip-rows variant has comparable latency, and AsyncLLM significantly outperforms that baseline in terms of accuracy.
We hope that additional latency measurements and text-based communication experiments helped alleviate your concerns. If you have any follow-up questions, please reply so we can continue our discussion. If you have no further concerns, please consider revising your score with these results in mind.
Initial response and pending experiments
Thank you for your review and feedback! We are working on experiments to better isolate shared memory and on additional latency measurements, expected within a week. Below, we describe what we are running so you may provide additional suggestions if you see fit.
We agree that a more detailed analysis of shared memory would improve our work. In our preliminary experiments, we found that basic asynchrony scenarios (e.g. asynchronous reasoning in Section 4.1) could, in principle, be implemented with an ordinary text communication between thinker and writer, but it dropped inference speed several times because the “writer” coroutine had to repeatedly re-encode thoughts. To address your concern, we are now re-running these experiments so they are comparable with our main setup. We will also attempt text-based Streaming Video Understanding that uses AsyncLLM coroutines but does not share memory blocks between coroutines.
Fortunately, we have some of the results already available from our VizDoom experiments. The numbers below come from the same data we used for Figure 4, measured on an H200 GPU. For comparison, we report both real-time latency and our original step-based measurements.
Table A: DoomHealthGathering latency, Qwen-3.6-35B-A3B, H200
Table B: DoomDeadlyCorridor latency, Qwen-3.6-35B-A3B, H200
These results suggest that a single-GPU AsyncLLM agent with a 35B MoE model can take 1-2 actions per second, depending on the environment. These latencies only quantify the time between receiving a new observation (game image) and producing the action. The background thinking coroutine takes longer, but it does not block the agent from taking immediate actions. Please also note that the real-time latency heavily depends on hardware, i.e. a faster GPU or multiple GPUs with tensor-parallelism can accelerate both AsyncLLM and baselines.
For monitoring (Table 2), our evaluations used a pool of different cloud GPUs. This did not affect reward and forward passes statistics that we report in Section 4.4, but it would be inconsistent for latency comparisons. We will run these experiments and report them within the next week.
We hope that our reported and ongoing experiments address your concerns. If you had something different or more specific in mind, please share your vision in a follow-up comment so we can adjust our evaluation setup.