acceptodds
Under review as a conference paper at ICLR 2027

MessageFlow: Proactive Message-Aware Execution for Efficient On-Device Multi-Agent Systems

Abstract

On-device multi-agent systems (MAS) powered by large language models (LLMs) face substantial response latency from chains of dependent LLM inference and tool calls. Agent-centric execution defers prompt processing until an agent is invoked, even when parts of its input are available earlier. We observe that evolving workflow messages can form stable, contiguous prompt prefixes that are ready for computation before control-flow dependencies are satisfied. Meanwhile, tool waits and autoregressive decoding often leave GPU resources underutilized, creating opportunities to overlap this early computation with ongoing execution. We present MessageFlow, a proactive message-aware execution framework that exploits these opportunities while preserving topology-defined agent invocation semantics. MessageFlow combines message update semantics, producer–consumer dependencies, and prompt ordering to identify reusable prefixes. Its runtime incrementally maintains and prioritizes the resulting prefill workloads, while an opportunity-aware GPU scheduler executes them during idle and decode-only periods and adapts to foreground demands to limit contention. The resulting key–value states are reused at invocation, moving part of prompt processing off the critical path. Implemented on LangGraph and SGLang, MessageFlow is evaluated on three representative workflows, four locally deployed LLMs, and three GPU platforms. Compared with state-of-the-art MAS execution systems, MessageFlow achieves up to 1.60× end-to-end speedup (1.32× on average), while increasing average GPU utilization from 57.8% to 89.6%.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
LLM-based Multi-Agent Systems, Efficient Inference, Edge AI, Runtime Systems

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TmMgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MessageFlow accelerates local LLM multi-agent workflows by processing stable prompt prefixes before downstream agents are formally invoked. It tracks message updates, dependencies, and prompt ordering to identify reusable prefixes, incrementally computes their inference states, and reuses them at invocation. Its scheduler performs proactive prefill during GPU idle periods or alongside decoding through SM partitioning, pausing work when foreground prefill arrives. Implemented using LangGraph and SGLang, the system reports reduced response-start latency across several workflows, models, and devices. Further measurements of changing workloads, peak memory, and precomputation reuse under different message patterns would clarify its practical benefits, resource costs, and limitations.

Critical strengths
  1. Useful execution opportunity. Processing stable downstream inputs early exposes overlap beyond the parallelism available from agent invocation dependencies.

  2. Thoughtful scheduling design. Incremental dispatch, resource partitioning, and foreground adaptation help control interference from proactive prefill.

  3. Concrete implementation and evaluation. Real workflow experiments across multiple models and devices, together with component ablations, demonstrate practical latency improvements.

Critical weaknesses
  1. Evaluate performance under changing workloads. The scheduler already adapts between idle, decoding, and prefill states. However, the SM partition within decoding periods remains fixed for each model–device pair. Please evaluate different context lengths, decoding batch sizes, and levels of concurrent agent activity. Report workflow latency and foreground decoding slowdown, including tail behavior. The offline profiling target of at most 10% decode-speed loss should also be checked under these conditions. These measurements would clarify when the selected partition remains effective and when workload-dependent adjustment becomes necessary.

  2. Report peak GPU memory usage and its scaling. Proactive requests retain KV states before their agents are invoked, while suspended prefill also retains intermediate tensors. Although only one proactive workload executes at a time, multiple future requests can retain completed states. Please compare peak GPU memory usage against the baselines and identify the additional memory attributable to proactive execution. Vary context length and the number of pending agents, including cases approaching device capacity. The implementation includes cleanup, but measurements are needed to establish the resource cost and practical limits of this on-device optimization.

  3. Measure coverage, actual reuse, and invalidation across message patterns. Table 4 already reports substantial aggregate precomputation coverage for the evaluated workflows. Please extend this analysis to different prompt orderings and workloads with more frequent overwriting, reordering, or summarization. Report precomputation coverage, the fraction of precomputed tokens actually reused, invalidation or fallback rates, and discarded computation time. An early unstable field can prevent later content from becoming eligible, while changes to an already processed prefix can invalidate work. These measurements would clarify how strongly the benefit depends on stable, append-only inputs.

Questions for authors
  1. Does the selected SM partition maintain its latency benefit and decode-slowdown target as workload characteristics change?

  2. How does peak memory overhead scale with context length and the number of pending agents?

  3. How do coverage, actual reuse, and invalidation rates vary across message update patterns and prompt orderings?

3 comments

We thank for your careful reading and detailed, insightful comments. Hope our responses below address your concerns.

1. Robustness of Selected Partition

(1) Context lengths and decode batch sizes. We evaluate the robustness of the selected SM partition under varying context lengths and decode batch sizes on Gemma3 27B with an RTX A6000, as shown in Figure 8 and Appendix E of the revised manuscript. Although absolute decode throughput varies with these parameters, the relative decode slowdown remains largely stable, ranging from 7.61% to 8.96% across context lengths and 8.28% to 10.02% across batch sizes. These results demonstrate the robustness of the selected partition under varying workload characteristics.

(2) Concurrent agent activity. Agent concurrency is not a direct determinant. When any active agent requires foreground prefill, the scheduler enters the BUSY state and suspends proactive prefill. SM partitioning is only activated during DECODE_ONLY periods. In this case, concurrent agent activity is reflected in the decode batch size, whose impact is evaluated above.

(3) Real-world workflow analysis. We further measure per-agent decode latency and slowdown during Open Deep Research execution using A6000 and Gemma3-27B, as reported in the table below. Each column represents an agent invocation, where S denotes Supervisor and R denotes Researcher. The numbers indicate agent's invocation rounds. Despite variations in context lengths, decode lengths, and agent execution patterns, the observed slowdown is consistent with our profiling results, further demonstrating the robustness of SM partitioning in dynamic MAS workflows. Some agents exhibit particularly low slowdown because no eligible proactive prefill workload is available during their decoding.

MethodS1R1-1R1-2R1-3R1-4R1-5R2-1R2-2R2-3S2
LangGraph5.03s1.57s1.56s1.67s2.07s2.07s1.67s2.09s2.37s3.58s
MessageFlow5.09s1.58s1.71s1.83s2.27s2.27s1.83s2.28s2.60s3.90s
Speed Loss1.18%0.63%8.77%8.74%8.81%8.81%8.74%8.33%8.85%8.21%

2. Memory Overhead

We have provided detailed measurements of peak GPU memory usage and memory overhead in our common response, “Memory Overhead and Performance under Memory Constraints.”

Regarding the suggestion to vary context length and the number of pending agents to approach device capacity, directly controlling these characteristics in real-world MAS workflows is challenging due to uncontrollable agent behavior. Instead, we vary the GPU memory budget available to the workflow to simulate memory-constrained conditions approaching device capacity. Detailed results are presented in the “Performance under Memory Constraints” section of our common response.

Regarding scalability, MessageFlow inherently involves a memory–performance tradeoff. As context lengths and the number of pending agents increase, more KV states may need to be retained. However, MessageFlow enforces a configurable GPU memory budget, preventing unbounded memory growth. With a larger memory budget, more proactive KV states can be retained and reused, potentially enabling greater acceleration.

3. Adaptation to Message Updates

First, MessageFlow adopts a relatively conservative strategy: it performs proactive prefill only on prefixes identified as stable, rather than speculatively computing potentially unstable content. Therefore, under the supported message semantics, previously computed prefixes are not invalidated by subsequent valid message updates and can all be reused when the corresponding agent is invoked.

MessageFlow already supports OVERWRITE semantics, as described in Section 3.1. Specifically, an overwritten message becomes eligible for proactive prefill only after it is guaranteed that no agent can further modify it. We consider message reordering and summarization as forms of OVERWRITE, since they effectively replace existing message content.

We acknowledge that frequent overwriting, reordering, or summarization can delay prefix stabilization, reducing precomputation coverage and available overlap opportunities. However, such updates do not introduce wasted proactive computation, since MessageFlow defers prefill until the corresponding prefix becomes stable. Consequently, MessageFlow would approach conventional execution when few stable prefixes are available, rather than incurring substantial overhead from invalidated precomputation.

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end, with approximately one day remaining, we would like to ensure that we have adequately addressed all your concerns. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we would be happy to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper. We wish you a wonderful day!

AuthorsOpenReview ↗

Kind Reminder

Dear Reviewer,

As the discussion period is nearing its end, with less than one day remaining, we sincerely hope you could take a moment to revisit our responses. We addressed your concerns early in the discussion period but have not yet received any follow-up feedback.

We would be extremely grateful if you could let us know whether your concerns have been adequately addressed.

Thank you so much for your time and consideration!

Reviewer_7t1DOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents MessageFlow, a proactive message-aware execution framework for reducing latency in on-device LLM-based multi-agent systems. The key insight is that an agent's prompt computation can begin before the agent is formally invoked, as parts of its input may already form stable and reusable prefixes. MessageFlow consists of three main components: (1) message-aware dependency tracking to identify stable prompt prefixes, (2) a runtime that maintains and prioritizes incremental prefill tasks, and (3) an opportunity-aware GPU scheduler that exploits idle and decode-only periods while minimizing interference with foreground execution. The system is implemented on LangGraph and SGLang and evaluated across three multi-agent workflows, four LLMs, and multiple GPU platforms. Experiments show consistent reductions in response-start latency, with an average speedup of 1.37× over LangGraph. The paper also provides ablation studies, GPU utilization analysis, and correctness discussions for cache reuse.

Critical strengths

S1. A practical and meaningful extension of proactive prefill for multi-agent systems.

The paper makes a useful distinction between agent invocation dependencies and prompt computation dependencies, allowing stable input prefixes to be processed before downstream agents are formally invoked. Compared with prior approaches such as FlashAgents, MessageFlow explicitly handles evolving workflow messages, including tool outputs, which can expose additional opportunities for computation overlap. The design is well motivated by realistic multi-agent execution patterns, and the discussion of prefix validation and correctness strengthens its technical foundation.

S2. Solid system implementation and encouraging experimental results.

The evaluation covers multiple workflows, models, and hardware platforms, with consistent latency improvements over the tested baselines, including optimized alternatives. The ablation studies further demonstrate that scheduling decisions, GPU resource partitioning, and foreground adaptation contribute meaningfully to performance. These results provide convincing evidence that MessageFlow is more than a straightforward application of prefix caching.

Critical weaknesses

W1. The independent novelty and contribution relative to prior proactive prefill methods could be better established.

Overlapping upstream generation with downstream prefill has already been explored by FlashAgents, while other systems such as KVFlow and Pythia investigate workflow-aware caching and execution optimization. Although MessageFlow extends these ideas to dynamic workflow messages and GPU scheduling opportunities, the evaluation does not sufficiently isolate the benefits of each new mechanism. In particular, it remains unclear whether a relatively simple extension of FlashAgents to support tool-generated messages and idle-period prefill could achieve much of the reported improvement. A more direct comparison or clearer attribution of the performance gains would strengthen the novelty claim.

W2. The evaluation leaves some uncertainty about practical end-to-end benefits and resource overhead.

The primary latency metric measures the time until the first token of the final response, rather than completion of the entire workflow output. For applications such as deep research, where the final response can be lengthy, the overall completion-time improvement may be smaller than the reported speedup. Additionally, the main latency experiments use relatively limited task samples, and samples with divergent execution traces are excluded without a detailed breakdown. The additional GPU memory requirements and potentially wasted computation from unused or invalidated prefixes are also not fully characterized. These factors matter particularly for resource-constrained on-device deployments.

Questions for authors

Q1. Novelty and comparison with FlashAgents.

Could the authors clarify how much of MessageFlow's performance improvement comes from handling dynamic tool messages versus its GPU scheduling mechanisms? Would extending FlashAgents with tool-message-triggered and idle-period prefill achieve similar benefits? What are the main technical challenges that distinguish MessageFlow from such an extension?

Q2. Full workflow completion latency.

How do the reported response-start speedups translate into full workflow completion latency, particularly for tasks with long final outputs such as Deep Research? A latency breakdown or additional measurements using existing workloads would help clarify the practical impact.

Q3. Resource overhead.

What are the additional GPU memory costs of maintaining proactive inference states? How much computation is wasted due to unused branches, invalidated prefixes, or canceled tasks? Could these overheads limit scalability in more complex workflows or memory-constrained environments?

3 comments

We thank for your careful reading and detailed, insightful comments. Hope our responses below address your concerns.

1. Novelty and comparison with FlashAgents.

(1) Sources of performance improvement. MessageFlow's performance gains arise from two complementary mechanisms: identifying precomputable stable prefixes from evolving messages and opportunity-aware GPU scheduling that overlaps proactive prefill with GPU bubbles. These mechanisms are tightly coupled: prefix identification determines what can be safely precomputed, while GPU scheduling determines when and how such computation can be executed without interfering with foreground requests. Consequently, their performance contributions cannot be simply separated or added independently. As shown in Table 5, disabling opportunity-aware scheduling increases Deep Research latency from 85.38 s to 127.77 s, even exceeding LangGraph (125.25 s) by 2.52 s.

(2) FlashAgents. We agree that extending FlashAgents with tool-message-triggered and idle-period prefill could improve its performance. However, we believe such extensions would not achieve benefits comparable to MessageFlow. First, FlashAgents employs batch-level prefill–decode overlap, similar to mixed batching, which provides limited benefits in our experiments (as discussed in Figure 3). This limitation motivates our SM partitioning design. As shown in Table 5 (w/o Resource Partitioning), replacing SM partitioning with mixed batching achieves only a 1.05× speedup on Deep Research, leaving substantial precomputation opportunities underexploited. Second, FlashAgents primarily focuses on sequential token streaming between agents and lacks a systematic abstraction for dynamic message dependencies and update semantics. In complex workflows such as Deep Research, downstream prompts aggregate messages from multiple agents and tools, which may undergo append or overwrite operations. Simply extending FlashAgents to handle tool-generated messages could therefore introduce challenges in message management and maintaining token-level prefix correctness.

2. Full workflow completion latency

We have additionally evaluated MAS end-to-end latency and compared it with MAS TTFT, with detailed results reported in Section 4.8 and Figure 6.

As expected, for long-output workloads such as Deep Research, the end-to-end speedup is lower than the MAS TTFT speedup (1.17× vs. 1.47×). In contrast, short-output workloads such as LLMCompiler and PrimoAgent are less affected, still achieving speedups of 1.41× and 1.60×, respectively.

Nevertheless, we consider MAS TTFT more relevant to interactive responsiveness, as it directly reflects user waiting time. Moreover, MessageFlow does not affect the final agent's decoding speed. Even Gemma3 27B, a relatively large model for on-device deployment, achieves over 10 tokens/s on our devices, exceeding the average human reading speed (~5.3 tokens/s).

3. Resource overhead

We have provided detailed measurements and analysis of MessageFlow's GPU memory overhead and performance under memory-constrained environments in our common response, “Memory Overhead and Performance under Memory Constraints”. Please refer to that response for details.

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end, with approximately one day remaining, we would like to ensure that we have adequately addressed all your concerns. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we would be happy to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper. We wish you a wonderful day!

Thank you for your rebuttal. I have no further questions!

Reviewer_NFmCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MessageFlow accelerates locally deployed multi-agent workflows by prefilling stable portions of downstream prompts before formal agent invocation. It combines explicit message-update semantics, incremental prefix tracking, workload prioritization, and opportunity-aware GPU scheduling. Experiments across three workflows, four models, and three GPU platforms report lower response-start latency, supported by scheduling ablations and numerical-quality checks.

Critical strengths

The paper identifies a useful distinction between readiness for agent invocation and readiness for partial prompt computation. Its implementation connects this observation to practical scheduling mechanisms rather than assuming that proactive execution is always beneficial. The scheduling ablations demonstrate the importance of controlling contention, and the cross-device experiments, perplexity comparisons, and downstream F1 evaluation provide meaningful supporting evidence.

Critical weaknesses

The performance evaluation does not fully establish the claimed end-to-end benefit. Section 4.1 measures latency only until the first token of the final response, whereas the abstract and conclusion describe end-to-end workflow speedups. Full completion latency and foreground decode latency are needed to assess whether work is accelerated overall. Moreover, Section 4.5 acknowledges nonidentical generated outputs. Differences in tool calls, loop counts, and response lengths could therefore affect the latency comparison. Similar aggregate HotpotQA F1 does not establish equivalent computational workloads, and the main text provides no corresponding quality evaluation for the research and trading workflows. The safety and resource assumptions are insufficiently characterized. APPEND semantics preserve committed message content, but do not by themselves guarantee an unchanged token prefix after tokenization and prompt rendering. The main text does not sufficiently explain how this condition is checked or how mismatches are handled. Proactive execution also retains KV states for agents that may be invoked much later or never selected by conditional routing, yet peak memory overhead, unused computation, and cache-pressure behavior are not quantified. These omissions matter particularly for the claimed on-device setting, where limited memory can offset the benefits of moving computation earlier.

Questions for authors

Could you report full workflow completion latency, foreground decode latency, and latency variability? Can fixed tool responses or trace-replay experiments confirm that the gains remain when competing systems perform equivalent work? How is token-level prefix stability established under text appends, chat-template rendering, and context truncation? Is the cached prefix checked against the actual invocation prompt, and what happens when they differ? What are the peak KV-cache overhead and the fraction of proactive computation ultimately unused? How does the runtime handle memory pressure, conditional branches, and repeated agent invocations? Could you apply the same opportunity-aware scheduler to Static or FlashAgents to separate scheduling gains from message-awareness gains? How sensitive are the results to prefill-unit size and the profiled SM partition ratios? I would consider raising my rating if the responses establish the full latency benefit under controlled workloads and clarify the correctness and memory-management boundaries.

4 comments

We thank for your careful reading and detailed, insightful comments. Hope our responses below address your concerns.

1. Full Workflow Completion Latency

We additionally evaluated full workflow completion latency (MAS end-to-end latency), as reported in Figure 6 and Section 4.8.

MessageFlow reduces completion latency across all three workloads, achieving 1.41× and 1.60× speedups for LLMCompiler and PrimoAgent, whose final outputs are under 100 tokens. For Deep Research (1,000–3,000 output tokens), the speedup drops from 1.47× (MAS TTFT) to 1.17× (end-to-end), as MessageFlow accelerates prefill rather than decoding.

Nevertheless, MAS TTFT better reflects interactive responsiveness by measuring user waiting time. MessageFlow does not affect decoding speed, and even Gemma3 27B achieves over 10 tokens/s on our devices, exceeding average human reading speed (~5.3 tokens/s).

Finally, we have revised the manuscript to consistently use MAS TTFT for response-start latency, reserving end-to-end latency for full workflow completion latency to avoid ambiguity.

2. Workload Equivalence and Comparison Fairness

We agree that numerical differences from GPU scheduling strategies may affect agent outputs and computational workloads.

To ensure fair comparisons, we exclude samples with different agent execution paths (e.g., loop counts) from latency evaluation, as these rare cases can substantially affect latency. We also replay fixed tool responses to eliminate network variability.

Regarding response lengths, we compare the Planner and Joiner outputs in LLMCompiler across 500 samples per model, as shown below. Across all models, output lengths differ in only 0.6–5.6% of samples, with mean absolute differences below 0.51 tokens. These negligible variations have little impact on measured latency, supporting the fairness of our comparisons.

ModelPlanner Diff. Rate (%)Planner Mean Abs. Diff. (tokens)Joiner Diff. Rate (%)Joiner Mean Abs. Diff. (tokens)
Gemma3 12B1.40%0.092.20%0.18
Phi4 14B3.40%0.485.60%0.51
Gemma3 27B0.60%0.041.40%0.06
Qwen3.5 27B2.00%0.223.60%0.32

3. Prefix Stability Assurance

We agree that APPEND semantics alone do not guarantee token-level prefix stability. To address this, MessageFlow already employs conservative prefix identification and exact token-level validation, as detailed in Appendix C.

MessageFlow's APPEND semantics guarantee monotonic growth of message content at the text level. However, this does not necessarily ensure token-level stability, as tokenization may change near append boundaries while most prefix tokens remain unchanged. Before each incremental proactive prefill, we render and tokenize the updated prefix, identify its longest common prefix (LCP) with the previously computed tokens, and retain the corresponding cache. Starting from the LCP boundary, we recompute the mismatched tokens and prefill the newly available tokens. At formal invocation, we similarly validate the cached tokens against the final prompt and reuse only the inference states corresponding to their LCP, ensuring correct KV cache reuse.

For chat-template rendering, we omit trailing boundary tokens of open APPEND regions, as they belong to the suffix rather than the stable prefix. Our current implementation does not support proactive prefill across context-truncation events.

4. Memory Overhead

For memory overhead and performance under memory pressure, please refer to our common response, “Memory Overhead and Performance under Memory Constraints”.

For repeated agent invocations, MessageFlow explicitly tracks agent lifetimes and incrementally updates their safe prefixes as workflow messages evolve. At each invocation, retained KV states are reused after token-level validation. Once an agent is confirmed to have no future invocations, its retained KV cache is released.

For conditional branches, MessageFlow treats them as potential execution paths and allows proactive prefill of their safe prefixes, while prioritizing deterministic paths to reduce speculative computation overhead. Unselected branches are removed from the proactive queue, and their retained KV states are released once they are confirmed unreachable.

5. Shared Opportunity-Aware Scheduling

In fact, Static, FlashAgents, and MessageFlow already share the same opportunity-aware GPU scheduler in our evaluation. Their key difference lies in proactive workload identification: Static and FlashAgents expose fewer prefill opportunities, whereas MessageFlow identifies additional stable prefixes from evolving workflow messages. In contrast, LangGraph uses the default SGLang backend without proactive scheduling.

We apologize for not clarifying this setup in the original submission and have now explicitly explained it in Appendix B (Baselines).

Dear Reviewer,

I hope this message finds you well. As the discussion period is nearing its end, with approximately one day remaining, we would like to ensure that we have adequately addressed all your concerns. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we would be happy to address any remaining issues to improve our work.

Thank you for your time and effort in reviewing our paper. We wish you a wonderful day!

AuthorsOpenReview ↗

Kind Reminder

Dear Reviewer,

As the discussion period is nearing its end, with less than one day remaining, we sincerely hope you could take a moment to revisit our responses. We addressed your concerns early in the discussion period but have not yet received any follow-up feedback.

We would be extremely grateful if you could let us know whether your concerns have been adequately addressed.

Thank you so much for your time and consideration!

Thank you to the authors for the additional experimental results and clarifications, which address my questions. I maintain my positive assessment and original scores.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Memory Overhead and Performance under Memory Constraints.

We thank the reviewers for raising important concerns about the memory overhead of proactive execution and its performance under memory constraints. To address these concerns, we conducted additional experiments, with detailed results included in Appendix H.

Memory Overhead. We first measure GPU memory usage over time for Open Deep Research with Gemma3-27B on an NVIDIA RTX A6000, as shown in Figure 10. Despite retaining additional KV states for proactive execution, MessageFlow exhibits a lower peak memory footprint than LangGraph (36.91 GiB vs. 42.22 GiB), a reduction of 5.31 GiB (12.6%).

This difference stems from their KV cache management policies. LangGraph relies on SGLang's default radix caching, which retains KV states from completed requests for potential reuse and evicts less recently used prefixes only when cache capacity becomes insufficient. This general-purpose policy relies on temporal locality rather than agent lifetimes to manage KV cache reuse. In contrast, MessageFlow leverages MAS execution semantics to explicitly manage KV cache lifetimes, proactively releasing KV states once the agents are no longer needed (Section 3.2). This targeted reclamation offsets the additional memory required for proactive execution, resulting in a lower overall peak memory footprint in the evaluated workload.

Performance under Memory Constraints. We further evaluate MessageFlow under different GPU memory budgets, with the results presented in the table below and Figure 11. The percentages indicate the proportion of total GPU memory allocated to the LLM server. Proactive execution inherently involves a memory–performance tradeoff: a larger KV cache capacity allows more proactive computation to be retained and reused, while a smaller memory budget limits proactive execution opportunities and may reduce performance gains. As shown in the table, MessageFlow's speedup gradually decreases as the GPU memory allocation ratio decreases, reaching 1.131× at 65%.

LangGraphMessageFlow+ KV offloadingMessageFlow SpeedUp+ Offloading SpeedUp
65%125.83s111.29s93.93s1.131×1.340×
70%125.81s107.10s92.41s1.175×1.361×
75%125.39s98.71s90.89s1.270×1.380×
80%125.94s94.67s88.40s1.330×1.425×
85%125.75s90.18s85.59s1.394×1.469×
90%125.99s85.60s85.52s1.472×1.473×

Importantly, this degradation can be mitigated through KV cache offloading. We implement a lightweight CPU offloading mechanism that transfers proactive KV states to host memory under GPU memory pressure and prefetches them before the corresponding agents execute. With offloading, MessageFlow's speedup at 65% GPU memory allocation improves from 1.131× to 1.340×. These results demonstrate that MessageFlow can benefit from hierarchical KV cache management, preserving substantial acceleration even under constrained GPU memory budgets.

Dear Reviewer,

As the discussion period is approaching its end, we would like to kindly follow up on our responses to your comments. We hope our clarifications have adequately addressed your concerns. If you have any remaining questions or suggestions, we would greatly appreciate your feedback and would be happy to provide further clarification.

Thank you again for your time and effort in reviewing our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.