acceptodds
Under review as a conference paper at ICLR 2027

LLMs are General Asynchronous Agents

Abstract

Modern LLMs are increasingly capable as autonomous agents, but they follow sequential interaction cycles: read, think, reply or call tools, repeat. Many real-world use cases are not sequential: voice assistants, embodied agents, and monitoring systems receive new inputs while they think or perform another task. Modern LLMs address this with specialized architectures for voice interaction and video streams, VLAs for robot control, asynchronous tool calling for API usage, and others. In this work, we generalize from different asynchronous tasks to general asynchronous agents that can adapt to different types of concurrency. To achieve this, we develop an asynchronous LLM framework that lets users (or the agents themselves) define inference coroutines with overlapping memory states. We showcase that Qwen 3.x models are capable of asynchronous operation for streaming video understanding, videogames, and monitoring, without task-specific training.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLMs, Asynchronous, asyncio, Inference, Training-free, agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Mm3mOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents AsyncLLM, a training-free framework for concurrent LLM inference with shared memory. The extensions to hybrid and multimodal models are technically useful, and the applications demonstrate feasibility. However, the experimental evidence does not sufficiently isolate the method’s benefits.

Critical strengths
  1. Useful technical contribution. The composable cache framework extends shared-memory concurrent inference to hybrid and multimodal models through Gated DeltaNet state composition and multimodal positional handling, exposing these capabilities through a unified programming interface.

  2. Broad applicability without task-specific training. Demonstrations spanning asynchronous text or image updates, streaming video, videogames, and system monitoring provide meaningful evidence that existing models can support diverse asynchronous workflows within the same framework.

Critical weaknesses
  1. The contribution of shared memory is not isolated. Table 1 compares Qwen-based AsyncLLM with Mage-VL, changing both the underlying model and inference framework. The same-model comparisons in games and monitoring compare different agent designs without isolating shared memory. A matched asynchronous baseline using ordinary text communication, with the same model and coroutine structure, is needed to establish what specifically benefits from the proposed mechanism.

  2. The responsiveness claims need direct latency measurements. Figure 4 and Table 2 measure delay using forward-pass counts, treating multiple tokens generated in one batch as one pass. However, pass duration varies with concurrency. The video throughput results establish processing capacity, but do not establish observation-to-action latency in games or monitoring. Please report wall-clock reward or accuracy versus response-latency curves on fixed hardware, including total inference cost.

Questions for authors

Please check the Weaknesses section.

6 comments
AuthorsOpenReview ↗

Initial response and pending experiments

Thank you for your review and feedback! We are working on experiments to better isolate shared memory and on additional latency measurements, expected within a week. Below, we describe what we are running so you may provide additional suggestions if you see fit.

The contribution of shared memory is not isolated. Table 1 compares Qwen-based AsyncLLM with Mage-VL, changing both the underlying model and inference framework. The same-model comparisons in games and monitoring compare different agent designs without isolating shared memory. A matched asynchronous baseline using ordinary text communication, with the same model and coroutine structure, is needed to establish what specifically benefits from the proposed mechanism.

We agree that a more detailed analysis of shared memory would improve our work. In our preliminary experiments, we found that basic asynchrony scenarios (e.g. asynchronous reasoning in Section 4.1) could, in principle, be implemented with an ordinary text communication between thinker and writer, but it dropped inference speed several times because the “writer” coroutine had to repeatedly re-encode thoughts. To address your concern, we are now re-running these experiments so they are comparable with our main setup. We will also attempt text-based Streaming Video Understanding that uses AsyncLLM coroutines but does not share memory blocks between coroutines.

The responsiveness claims need direct latency measurements. Figure 4 and Table 2 measure delay using forward-pass counts, treating multiple tokens generated in one batch as one pass. However, pass duration varies with concurrency. The video throughput results establish processing capacity, but do not establish observation-to-action latency in games or monitoring. Please report wall-clock reward or accuracy versus response-latency curves on fixed hardware, including total inference cost.

Fortunately, we have some of the results already available from our VizDoom experiments. The numbers below come from the same data we used for Figure 4, measured on an H200 GPU. For comparison, we report both real-time latency and our original step-based measurements.

Table A: DoomHealthGathering latency, Qwen-3.6-35B-A3B, H200

MethodRewardForward / actionSeconds / action
Probe508.910.06
Sequential575.81132.764.59
AsyncLLM (main)573.4517.260.58
AsyncLLM (self-defined)589.7841.901.46

Table B: DoomDeadlyCorridor latency, Qwen-3.6-35B-A3B, H200

MethodRewardForward / actionSeconds / action
Probe184.4910.07
Sequential340.10116.884.25
AsyncLLM (main)362.2424.570.96
AsyncLLM (self-defined)297.8849.662.04

These results suggest that a single-GPU AsyncLLM agent with a 35B MoE model can take 1-2 actions per second, depending on the environment. These latencies only quantify the time between receiving a new observation (game image) and producing the action. The background thinking coroutine takes longer, but it does not block the agent from taking immediate actions. Please also note that the real-time latency heavily depends on hardware, i.e. a faster GPU or multiple GPUs with tensor-parallelism can accelerate both AsyncLLM and baselines.

For monitoring (Table 2), our evaluations used a pool of different cloud GPUs. This did not affect reward and forward passes statistics that we report in Section 4.4, but it would be inconsistent for latency comparisons. We will run these experiments and report them within the next week.

We hope that our reported and ongoing experiments address your concerns. If you had something different or more specific in mind, please share your vision in a follow-up comment so we can adjust our evaluation setup.

AuthorsOpenReview ↗

Text-based communication and latency (follow-up 1/2)

As we promised earlier, we implemented text-based communication in several environments to benchmark effectiveness and latency.

Text-based Asynchronous Reasoning

To better decouple different moving parts, we begin with text-based communication in the setup from Section 4.1 with k=0, i.e. solving a fully defined problem from MATH-500 on H200 with Qwen3.5-27B. To run the extra compute-heavy experiments on time, we test this section using 100 random samples out of MATH-500 since the differences we observed are large enough. We start with the same asynchronous reasoning setup where the ‘writer’ can write partial responses while the ‘thinker’ is still reasoning about the problem, thus solving the problem faster.

We first attempted several straightforward ways to support text-based communication: letting two agents message each other using tools or running our standard memory pipeline with re-encoding new text tokens that are accessible to the thinker/writer coroutines without sharing memory states. However, we found that both versions introduced significant real-time delays where every minor update to the memory state required processing tens of extra tokens.

To alleviate this, we implemented a modified protocol where the thinker coroutine proceeded normally, but the writer switched only pulled updates every 50 steps and ran standard inference between them. This improved baseline achieved significantly smaller performance drawdowns without compromising accuracy, though it is still significantly slower than shared cache.

Table C: Asynchronous Reasoning, 100 samples from MATH-500, Qwen3.5-27B, H200

SetupAccuracyMean latency
AsyncLLM (main, shared cache)94%72.98 s
Text-based communication as-is95%6194.96 s
Text-based every 50 writer steps94%257.72 s

To summarize, asynchronous agents with text-based communication obtain similar accuracy scores, but take significantly longer per sample due to the additional computations for passing and re-encoding tokens. Straightforward text-based conversion resulted in very long delays (~1.5 hours) that could be reduced by redesigning the agent to communicate less often.

Text-based Streaming Video Understanding

For streaming video understanding, naive token-based communication without changing the agent architecture likewise ran several times slower on test problems and we would be unable to evaluate it on the entire dataset within our compute budget. However, we got significantly better results by redesigning the text-based agent using AsyncLLM that only passes messages when the probe found a description-worthy event.

Table D: improved text-based agent on ProactiveVideoQA, Qwen3-9B, 1x H200.

SetupWEBEGOTVVADFrame latencyWall time
AsyncLLM (main, shared cache)49.356.363.835.80.78s6.5s
AsyncLLM (text-based, optimized)47.256.760.532.341.23s7.6s

In Table D above, we report accuracy on ProactiveVideoQA subsets and two latency metrics. Frame latency is how long it takes to process a single video frame, from receiving the image to deciding whether or not it has a new event to describe, including the time it takes to communicate the result. Wall time is the average time to process a StreamingVideoQA sample.

Overall, the optimized text-based agent has comparable accuracy scores across subsets, but takes ~58% longer to reason about a given frame and ~17% slower overall. Note that wall time is less affected by video delays because text-based AsyncLLM can still schedule multiple parallel frames into the same batch, with denser workload compensating for delays.

Text-based VizDoom

Similarly, we only evaluate the optimized agent where the short-term action coroutine only re-encodes context after taking its action, which significantly reduces GPU latency but introduces slight thought-action delays.

Table A2: DoomHealthGathering latency, Qwen-3.6-35B-A3B, H200

MethodRewardForward / actionSeconds / action
Probe508.910.06
Sequential575.81132.764.59
AsyncLLM (main)573.4517.260.58
AsyncLLM (self-defined)589.7841.901.46
AsyncLLM (text-based)573.8116.952.20

Table B2: DoomDeadlyCorridor latency, Qwen-3.6-35B-A3B, H200

MethodRewardForward / actionSeconds / action
Probe184.4910.07
Sequential340.10116.884.25
AsyncLLM (main)362.2424.570.96
AsyncLLM (self-defined)297.8849.662.04
AsyncLLM (text-based)354.819.283.79

We evaluate these text-based agents using the same setups as in our previous response and report them in tables A2, B2, at the bottom row. Both text-based agents got reasonable scores, but they had to play a much slower version of the game to match with their decision-making speed. Additionally, the thought-action delay from optimized reencoding appears to have had some effect on the ‘shooter’ scenario in DoomDeadlyCorridor.

AuthorsOpenReview ↗

DevOpsGym Monitoring (follow-up 2/2)

Your review requested additional latency benchmarks for Figure 4 (VizDoom) and Table 2 (Monitoring). Earlier, we reported VizDoom latency in our initial response (Tables A, B) and promised to re-run our monitoring evaluation on the same GPU type so you can compare real-world latency, and we report these results now. We measure the average total time it takes to process a full scenario if the events are fed as soon as the agent is ready to ingest them.

Table E: DevOpsGym Monitoring, Qwen3.6-35B-A3B Monitoring, 1 H200

SetupAccuracy (%)Wall time, s.
Q3.6-35B-A3B61.76204.47
+ early answer47.06162.58
+ skip rows26.4790.58
AsyncLLM55.8897.56

In the table above, the relative speed-up is slightly (<10%) worse in latency than it was in terms of forward passes due to slight parallelism overhead, but AsyncLLM still maintains a strong advantage over sequential baselines. Only the skip-rows variant has comparable latency, and AsyncLLM significantly outperforms that baseline in terms of accuracy.

We hope that additional latency measurements and text-based communication experiments helped alleviate your concerns. If you have any follow-up questions, please reply so we can continue our discussion. If you have no further concerns, please consider revising your score with these results in mind.

Reviewer_eUSbOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper asks whether LLM agents can be made generally asynchronous without task-specific training, instead of each application such as voice, video, or robotics building its own concurrent architecture. The authors propose AsyncLLM, a framework that maps the asyncio async/await model onto LLM inference. An agent is a set of coroutines, each writing to its own cache block of KV and recurrent states, and coroutines see each other's progress in real time through ordered cache views over shared blocks. Technically, the paper extends prior concurrent attention work to hybrid models by composing Gated Delta Net states as per-block affine transitions, and to multimodal models via MRoPE corrections. An engine built on mini-SGLang batches concurrent coroutines on one GPU. Experiments cover mid-reasoning text and image corrections, streaming video understanding on SoccerNet and ProactiveVideoQA where training-free agents beat the purpose-trained Mage-VL, ViZDoom game playing with faster reactions than a sequential agent, and system log monitoring with comparable accuracy at much lower latency.

Critical strengths
  1. A timely and well framed problem. The paper identifies a real commonality across voice assistants, streaming video, embodied agents, and monitoring, and the async/await formulation with shared memory views is a clean and reusable abstraction.
  2. Solid technical contributions. The block-wise affine composition for linear attention and the MRoPE handling are correct, efficient in the number of blocks, and cover most modern hybrid architectures, as the appendix discussion of attention variants shows.
  3. Broad demonstrations. Four qualitatively different applications with the same framework, honest reporting of limits such as the unreliable self-defining agents, and a useful latency analysis all support the feasibility claim.
Critical weaknesses
  1. The per-application comparisons are thin. Streaming video is compared only against Mage-VL with hyperparameters tuned by the authors themselves, the game and monitoring experiments compare only against same-model sequential baselines, and the monitoring subset has 34 tasks where the asynchronous agent is slightly less accurate than the sequential one. The results show feasibility more than superiority.
  2. All experiments use Qwen3.x models from one family, and the shared memory state under concurrent writes is not equivalent to any sequential inference. There is little analysis of when this deviation hurts, so it is unclear how much the generality claim depends on this model family.
Questions for authors
  1. When several coroutines update blocks concurrently while watching each other, the composed state has no sequential equivalent. Did you observe accuracy degradation as the number of concurrently updated blocks grows, and is there a regime where the shared state breaks down?
  2. How sensitive are the results to the design of cache views, such as the order of blocks or which blocks a coroutine can see?
  3. For the self-defining agents in Appendix J, what failed on DeadlyCorridor, and what would it take to make agent-defined coroutines reliable?
5 comments
AuthorsOpenReview ↗

Initial response and pending experiments

Thank you for a thoughtful and encouraging review! Our main motivation for writing this paper was indeed about finding commonalities and reusable building blocks across different applications of asynchronous agents, and we are glad that your review highlights them. Below, we address the concerns and questions raised in your review, and describe the experiments we are currently conducting based on your feedback.

(Weakness 1) The results show feasibility more than superiority.

Yes, precisely. Our results show that general LLMs can work as asynchronous agents across different use cases, but not that they beat every task-specific model. While it does outperform some specialized baselines, the main advantage of AsyncLLM agents is that they can adapt to new use cases by changing the agent definition, which is easier than training the weights. For instance, in our VizDoom experiments (S4.3), it should be possible to train an agent that solves one specific scenario better. The advantage of using AsyncLLM is in that they could adapt to a completely different environment by simply changing the coroutine definitions.

(Weakness 2) All experiments use Qwen3.x models from one family, and the shared memory state under concurrent writes is not equivalent to any sequential inference.

We agree that additional models would further improve our work and are working on supporting another architecture outside of the Qwen family, we will report our findings in a follow-up message within 7 days. For concurrent writes, please see the next paragraph.

(Question 1) Did you observe accuracy degradation as the number of concurrently updated blocks grows, and is there a regime where the shared state breaks down?

When experimenting with Streaming Video Understanding and VizDoom agents, we initially considered more complicated agent designs with parallel verifiers, strategic vs. tactical thinkers, and other subtasks. There, we found no signs of degradation for 4-5 concurrent coroutines, but we ultimately found that a simpler agent design can perform just as well for these tasks.

To stress-test AsyncLLM with more concurrency, we will conduct an experiment where multiple concurrent sub-thinkers are working on a shared reasoning task and see each other’s cache as it is updated. This setup will allow us to vary the number of agents without confounding changes to the agent architecture. We will report the outcome within a week of writing this message.

(Question 2) How sensitive are the results to the design of cache views, such as the order of blocks or which blocks a coroutine can see?

There is no single answer here, so we simply report our findings. When experimenting with cache blocks, we found that changing the order does not “break” the agent, but can affect performance.

In Streaming Video Understanding, we found that encoding a pair of images as one “turn” in the chat template allowed the agent to better detect video events than if we fed the agent one frame at a time. This was particularly significant for the smaller Qwen-3.5-9B agent, but larger models also showed some improvement. This is why our agents use pair frames (L2029).

In VizDoom environments, we found that if the action-taking coroutine starts occasionally hallucinating image details if it sees a lot of additional text beside the image and the instructions, especially if it sees previous image descriptions after the latest image. Though we did not know this at the time of submission, we later found a possibly related phenomenon in several recent works, where the VLM got worse at "seeing" when distracted by reasoning [1,2] or long context [3, 4].

(Question 3) For the self-defining agents in Appendix J, what failed on DeadlyCorridor, and what would it take to make agent-defined coroutines reliable?

We inspected the generated code and found that a coroutine responsible for ‘strategizing’ (apparently) only ran for 100 steps before it finished. The agent still performed better than the probe because, by then, it was already close enough to solve the problem using the rules the agent wrote in its ‘system prompt’.

We are now conducting experiments with more concurrent coroutines to answer your question about potential degradation, expected within 7 days from posting this message. If you had some other setup in mind or if you had any additional suggestions, please share them in a follow-up comment.

  • [1] Xu, Zhongxing, et al. "More thinking, less seeing? assessing amplified hallucination in multimodal reasoning models."
  • [2] Sun, Hai-Long, et al. "Mitigating visual forgetting via take-along visual conditioning for multi-modal long cot reasoning."
  • [3] Sharma, Aditya, Michael Saxon, and William Yang Wang. "Losing visual needles in image haystacks: Vision language models are easily distracted in short and long contexts."
  • [4] Liu, Ming, et al. "On the robustness of multimodal language model towards distractions."

I thank the authors for the honest and quick response. [W1]: I agree the main claim is generality, and the rebuttal makes this point clear. [W2]: Good to know. I look forward to the results on a non-Qwen model. [Q1]: The 4–5 coroutine observation is useful. I will wait for the follow-up. [Q2]: Thanks, these findings are helpful, especially the VizDoom hallucination issues. [Q3]: Thanks for checking the code. The 100-step strategist explains the failure, but the reliability question is still open. Overall, the response addresses my points well so far. I will maintain my score.

AuthorsOpenReview ↗

Additional model family (follow-up 1/2)

As we promised earlier, we conducted additional experiments to test an additional model family and evaluated AsyncLLM with more concurrently updated blocks.

All experiments use Qwen3.x models from one family, and the shared memory state under concurrent writes is not equivalent to any sequential inference. There is little analysis of when this deviation hurts, so it is unclear how much the generality claim depends on this model family.

To verify that AsyncLLM agents can be built around other model families, we integrated the Gemma 4 31B vision language model into the AsyncLLM framework. Gemma 4 uses slightly different positional encoding (p-RoPE with p=0.25p=0.25 and hybrid window attention, see Section 2 in the Gemma 4 technical report [1]), but it allows the cache manipulation strategies AsyncLLM needs to compose cache blocks in different coroutines.

We ran Gemma 4 31B in FP8 (H200 GPU) in the Asynchronous Reasoning setup from Section 4.1 (no async inputs, k=0) and on the first videogame setup from Section 4.3. In both cases, we had to change the special tokens in prompts as Gemma 4 uses a slightly different tokenization scheme, but we kept the instructions as is.

Table A: Gemma-4-31B FP8 AsyncReasoning, MATH-500, H200

MethodAccuracyAvg. forward passesAvg. generation time
AsyncLLM based on [2]496/500 (99.2%)1,274.5159.0 s
Native sequential thinking495/500 (99.0%)2,671.6309.3 s
Native without thinking489/500 (97.8%)679.377.5 s

For all methods, we let the agent reason for up to 16K inference steps or until it finishes its turn normally, then run the LLM judge to extract the answer and compare against the reference. The number of forward passes is the average number of times the LLM engine runs the model, where multiple coroutines can fit in the same batch. The average generation time is wall clock time from receiving the task to finishing generation.

Table B: Gemma-4-31B, DoomHealthGathering, 100 samples, H200

MethodRewardForward / actionSeconds / action
Probe494.3510.146
Sequential558.5889.1210.26
AsyncLLM565.4911.791.51

Our DoomHealthGathering uses the same configuration as in Section 4.3, except for Gemma’s different tokenization schema. Gemma-4-31B produces similar results to Qwen3.6-35B-A3B, but has slower wall clock time due to being a dense model. The results above show that AsyncLLM can work with the Gemma-4 model family. After the discussion, we will evaluate additional benchmarks for the final version of the paper.

  • [1] Gemma Team, et al. "Gemma 4 technical report." arXiv preprint arXiv:2607.02770 (2026).
  • [2] Yakushev, George, et al. "Asynchronous reasoning: Training-free interactive thinking llms." arXiv preprint arXiv:2512.10931 (2025).
Reviewer_R8czOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AsyncLLM, which uses async/await to organize concurrent inference and lets coroutines read one another's generation states through shared cache blocks. It extends shared-cache inference to hybrid-attention and multimodal models and automatically batches concurrent requests. Using Qwen models without task-specific fine-tuning, the authors evaluate mid-reasoning input updates, streaming video, ViZDoom, and system-log monitoring. The experiments show that the framework can process new inputs alongside background reasoning and improve responsiveness and task performance in some settings.

Critical strengths
  1. The technical contribution goes beyond coroutine orchestration. Block-wise composition of Gated Delta Network states and multimodal position handling extend shared-cache inference to modern hybrid models. This is the paper's clearest methodological contribution.
  2. One framework handles mid-reasoning input updates, video, games, and log monitoring without task-specific fine-tuning. Games and monitoring also include same-model sequential controls. Together, these results support the practical usefulness of a unified asynchronous interface.
Critical weaknesses
  1. The contributions of concurrency, shared caches, and task-specific probes have not been separated. Video primarily compares different models, making it particularly difficult to determine the additional benefit of shared caches over conventional asynchronous orchestration. This limits assessment of how necessary the core design is.
  2. The responsiveness comparison does not sufficiently control task quality. Monitoring agents answer earlier but are less accurate than the full-log baseline, while the main delay measure counts forward passes rather than actual waiting time. It therefore remains unclear how much benefit comes from concurrency versus reading fewer logs and deciding earlier.
Questions for authors
  1. With the same model and comparable compute budgets, how much does shared-cache inference improve task quality or response time over concurrent processing with text-based communication? This would help establish the value of the cache mechanism itself.
  2. Are the additional monitoring errors relative to the full-log baseline mainly caused by answers being emitted before critical logs arrive? Can existing per-task traces explain how these errors relate to actual response time?
4 comments
AuthorsOpenReview ↗

Initial response and pending experiments

Thank you for your feedback! We are glad that you highlight the generality of AsyncLLM as a critical strength, as it is the main reason we conducted this research. Below, we answer your questions and describe additional evaluations we are currently running to address your concerns.

(Weakness 1 / Question 1) With the same model and comparable compute budgets, how much does shared-cache inference improve task quality or response time over concurrent processing with text-based communication? This would help establish the value of the cache mechanism itself.

We agree that comparing shared cache and text-based communication can improve our work, and we describe our steps to address it below. The caveat with text-based communication is that our current agent design relies on shared cache. For instance, Section 4.1 response writer reads the chain-of-thought in real time after every token, and Section 4.3 VizDoom thinker instantly reacts to image updates. When converted to text-based agents, the agents slow down by over 3.5x (Doom) because their coroutines need to re-encode each other’s tokens on almost every inference step.

To provide a stronger baseline, we are currently experimenting with text-based communication where agents exchange messages less frequently to maintain low action latency. We are running these experiments for asynchronous reasoning, streaming video understanding, and VizDoom, and we hope to share our findings by November 16th AOE.

(Weakness 2) The responsiveness comparison does not sufficiently control task quality. Monitoring agents answer earlier but are less accurate than the full-log baseline, while the main delay measure counts forward passes rather than actual waiting time.

To address this concern, we are re-running our DevOpsGym Monitoring experiments to measure real-time delays on the same GPUs, by the same deadline as in the previous section.

(Question 2) Are the additional monitoring errors relative to the full-log baseline mainly caused by answers being emitted before critical logs arrive? Can existing per-task traces explain how these errors relate to actual response time?

We inspected the traces where Qwen 3.6 35B A3B provided the different answers between AsyncLLM and the full log baseline. We found 3 samples where the full baseline was correct, 1 sample where AsyncLLM was correct but the full baseline failed, 6 samples they gave different incorrect answers. In cases where baseline or AsyncLLM were correct, all except one had enough data to detect the problem correctly by the time AsyncLLM gave its answer. The remaining case was a memory leak that AsyncLLM agent stopped after ~6.5% events after hallucinating that there was an IO bottleneck, seemingly misinterpreting the disk logs.

To verify that the remaining cases are solvable from partial inputs, we ran Claude Opus 5.5 (API, pass@3) on excerpts with and verified that it gave the correct answer: it solved 2/3 samples where AsyncLLM failed but the full baseline was correct, 1/1 sample with the opposite, and 4/6 samples where both Qwen-3.6-35B-A3B agents failed. We also found that in most logs, the logs contain multiple repeated events (CPU spikes) or a gradual onset (memory/file leak). To summarize, there was indeed a premature answer, but more frequently, the agent had sufficient logs and failed because of reasoning errors.

We are now conducting additional experiments with text-based communication and monitoring latency and plan to report our results in a follow-up response by Monday, 16th (AOE). If you have any follow-up comments or if we misunderstood any of your feedback, please comment so we may adjust our evaluations to better address your concerns.

AuthorsOpenReview ↗

DevOpsGym Monitoring (follow-up 1/2)

These are our results for DevOpsGym Monitoring latency and text-based communication that we promised earlier. Please note that you and Reviewer Mm3m requested similar evaluations, so many of the additional results we report are from the same set experiments we conducted based on both your and reviewer Mm3m’s suggestions. We hope that additional latency measurements and text-based communication experiments helped alleviate your concerns. If you have any follow-up questions or suggestions, please respond below.

DevOpsGym Monitoring

Monitoring agents answer earlier but are less accurate than the full-log baseline, while the main delay measure counts forward passes rather than actual waiting time. It therefore remains unclear how much benefit comes from concurrency versus reading fewer logs and deciding earlier.

We re-run our evaluations on the same hardware to compare latency and provide an additional baseline with a faster but weaker model. In these experiments, we measure the average total time it takes to process a full scenario if the events are fed as soon as the agent is ready to ingest them. Additionally, we report a faster baseline with a faster Qwen3.5-4B model with full logs and without AsyncLLM.

Table A: DevOpsGym Monitoring, Qwen3.6-35B-A3B Monitoring, 1 H200

SetupAccuracy (%)Wall time, s.
Q3.6-35B-A3B61.76204.47
+ early answer47.06162.58
+ skip rows26.4790.58
AsyncLLM55.8897.56
Q3.5-9B47.06254.65
Q3.5-4B32.35147.23

In the table above, the relative speed-up is slightly (<10%) worse in latency than it was in terms of forward passes due to slight parallelism overhead, but AsyncLLM still maintains a strong advantage over sequential baselines. The 4B full baseline gives significant speed-ups, but the accuracy drops significantly compared to the asynchronous agent. Note that our other Qwen3.5-9B model is not faster than 35B-A3B because the former is a dense model and the latter is a mixture of experts.

AuthorsOpenReview ↗

Text-based communication and latency (follow-up 2/2)

Text-based Asynchronous Reasoning

With the same model and comparable compute budgets, how much does shared-cache inference improve task quality or response time over concurrent processing with text-based communication? This would help establish the value of the cache mechanism itself.

To better decouple different moving parts, we begin with text-based communication in the setup from Section 4.1 with k=0, i.e. solving a fully defined problem from MATH-500 on H200 with Qwen3.5-27B. To run the extra compute-heavy experiments on time, we test this section using 100 random samples out of MATH-500 since the differences we observed are large enough. We start with the same asynchronous reasoning setup where the ‘writer’ can write partial responses while the ‘thinker’ is still reasoning about the problem, thus solving the problem faster.

We first attempted several straightforward ways to support text-based communication: letting two agents message each other using tools or running our standard memory pipeline with re-encoding new text tokens that are accessible to the thinker/writer coroutines without sharing memory states. However, we found that both versions introduced significant real-time delays where every minor update to the memory state required processing tens of extra tokens.

To alleviate this, we implemented a modified protocol where the thinker coroutine proceeded normally, but the writer switched only pulled updates every 50 steps and ran standard inference between them. This improved baseline achieved significantly smaller performance drawdowns without compromising accuracy, though it is still significantly slower than shared cache.

Table B: Asynchronous Reasoning, 100 samples from MATH-500, Qwen3.5-27B, H200

SetupAccuracyMean latency
AsyncLLM (main, shared cache)94%72.98 s
Text-based communication as-is95%6194.96 s
Text-based every 50 writer steps94%257.72 s

To summarize, asynchronous agents with text-based communication obtain similar accuracy scores, but take significantly longer per sample due to the additional computations for passing and re-encoding tokens. Straightforward text-based conversion resulted in very long delays (~1.5 hours) that could be reduced by redesigning the agent to communicate less often.

Text-based Streaming Video Understanding

For streaming video understanding, naive token-based communication without changing the agent architecture likewise ran several times slower on test problems and we would be unable to evaluate it on the entire dataset within our compute budget. However, we got significantly better results by redesigning the text-based agent using AsyncLLM that only passes messages when the probe found a description-worthy event.

Table C: improved text-based agent on ProactiveVideoQA, Qwen3-9B, H200.

SetupWEBEGOTVVADFrame latencyWall time
AsyncLLM (main)49.356.363.835.80.78s6.5s
AsyncLLM (text-based)47.256.760.532.341.23s7.6s

In Table C above, we report accuracy on ProactiveVideoQA subsets and two latency metrics. Frame latency is how long it takes to process a single video frame, from receiving the image to deciding whether or not it has a new event to describe, including the time it takes to communicate the result. Wall time is the average time to process a StreamingVideoQA sample.

Overall, the optimized text-based agent has comparable accuracy scores across subsets, but takes ~58% longer to reason about a given frame and ~17% slower overall. Note that wall time is less affected by video delays because of AsyncLLM parallelism.

Text-based VizDoom

Similarly, we only evaluate the optimized agent where the short-term action coroutine only re-encodes context after taking its action, which significantly reduces GPU latency but introduces slight thought-action delays.

Table D: DoomHealthGathering, Qwen-3.6-35B-A3B, H200

MethodRewardForward / actionSeconds / action
Probe508.910.06
Sequential575.81132.764.59
AsyncLLM (main)573.4517.260.58
AsyncLLM (self-defined)589.7841.901.46
AsyncLLM (text-based)573.8116.952.20

Table E: DoomDeadlyCorridor, Qwen-3.6-35B-A3B, H200

MethodRewardForward / actionSeconds / action
Probe184.4910.07
Sequential340.10116.884.25
AsyncLLM (main)362.2424.570.96
AsyncLLM (self-defined)297.8849.662.04
AsyncLLM (text-based)354.819.283.79

We evaluate these text-based agents using the same setups as in our previous response and report them in tables D, E at the bottom row. Both text-based agents got reasonable scores, but they had to play a much slower version of the game to match with their decision-making speed. Additionally, the thought-action delay from optimized reencoding appears to have had some effect on the ‘shooter’ scenario in DoomDeadlyCorridor.

Thank you for your responses. I have decided to maintain my original score.

Comments on the paper

1 comment

Dear Reviewers,

The authors have responded to your questions. If you haven’t already done so, please review their responses and let them know whether your concerns have been addressed. Please use the discussion to raise any remaining concerns or follow-up questions.

Thank you for your time!

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.