- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MessageFlow accelerates local LLM multi-agent workflows by processing stable prompt prefixes before downstream agents are formally invoked. It tracks message updates, dependencies, and prompt ordering to identify reusable prefixes, incrementally computes their inference states, and reuses them at invocation. Its scheduler performs proactive prefill during GPU idle periods or alongside decoding through SM partitioning, pausing work when foreground prefill arrives. Implemented using LangGraph and SGLang, the system reports reduced response-start latency across several workflows, models, and devices. Further measurements of changing workloads, peak memory, and precomputation reuse under different message patterns would clarify its practical benefits, resource costs, and limitations.
Critical strengths
-
Useful execution opportunity. Processing stable downstream inputs early exposes overlap beyond the parallelism available from agent invocation dependencies.
-
Thoughtful scheduling design. Incremental dispatch, resource partitioning, and foreground adaptation help control interference from proactive prefill.
-
Concrete implementation and evaluation. Real workflow experiments across multiple models and devices, together with component ablations, demonstrate practical latency improvements.
Critical weaknesses
-
Evaluate performance under changing workloads. The scheduler already adapts between idle, decoding, and prefill states. However, the SM partition within decoding periods remains fixed for each model–device pair. Please evaluate different context lengths, decoding batch sizes, and levels of concurrent agent activity. Report workflow latency and foreground decoding slowdown, including tail behavior. The offline profiling target of at most 10% decode-speed loss should also be checked under these conditions. These measurements would clarify when the selected partition remains effective and when workload-dependent adjustment becomes necessary.
-
Report peak GPU memory usage and its scaling. Proactive requests retain KV states before their agents are invoked, while suspended prefill also retains intermediate tensors. Although only one proactive workload executes at a time, multiple future requests can retain completed states. Please compare peak GPU memory usage against the baselines and identify the additional memory attributable to proactive execution. Vary context length and the number of pending agents, including cases approaching device capacity. The implementation includes cleanup, but measurements are needed to establish the resource cost and practical limits of this on-device optimization.
-
Measure coverage, actual reuse, and invalidation across message patterns. Table 4 already reports substantial aggregate precomputation coverage for the evaluated workflows. Please extend this analysis to different prompt orderings and workloads with more frequent overwriting, reordering, or summarization. Report precomputation coverage, the fraction of precomputed tokens actually reused, invalidation or fallback rates, and discarded computation time. An early unstable field can prevent later content from becoming eligible, while changes to an already processed prefix can invalidate work. These measurements would clarify how strongly the benefit depends on stable, append-only inputs.
Questions for authors
-
Does the selected SM partition maintain its latency benefit and decode-slowdown target as workload characteristics change?
-
How does peak memory overhead scale with context length and the number of pending agents?
-
How do coverage, actual reuse, and invalidation rates vary across message update patterns and prompt orderings?
3 comments
Dear Reviewer,
I hope this message finds you well. As the discussion period is nearing its end, with approximately one day remaining, we would like to ensure that we have adequately addressed all your concerns. If there are any additional points or feedback you'd like us to consider, please let us know. Your insights are invaluable to us, and we would be happy to address any remaining issues to improve our work.
Thank you for your time and effort in reviewing our paper. We wish you a wonderful day!
Kind Reminder
Dear Reviewer,
As the discussion period is nearing its end, with less than one day remaining, we sincerely hope you could take a moment to revisit our responses. We addressed your concerns early in the discussion period but have not yet received any follow-up feedback.
We would be extremely grateful if you could let us know whether your concerns have been adequately addressed.
Thank you so much for your time and consideration!
We thank for your careful reading and detailed, insightful comments. Hope our responses below address your concerns.
1. Robustness of Selected Partition
(1) Context lengths and decode batch sizes. We evaluate the robustness of the selected SM partition under varying context lengths and decode batch sizes on Gemma3 27B with an RTX A6000, as shown in Figure 8 and Appendix E of the revised manuscript. Although absolute decode throughput varies with these parameters, the relative decode slowdown remains largely stable, ranging from 7.61% to 8.96% across context lengths and 8.28% to 10.02% across batch sizes. These results demonstrate the robustness of the selected partition under varying workload characteristics.
(2) Concurrent agent activity. Agent concurrency is not a direct determinant. When any active agent requires foreground prefill, the scheduler enters the BUSY state and suspends proactive prefill. SM partitioning is only activated during DECODE_ONLY periods. In this case, concurrent agent activity is reflected in the decode batch size, whose impact is evaluated above.
(3) Real-world workflow analysis. We further measure per-agent decode latency and slowdown during Open Deep Research execution using A6000 and Gemma3-27B, as reported in the table below. Each column represents an agent invocation, where S denotes Supervisor and R denotes Researcher. The numbers indicate agent's invocation rounds. Despite variations in context lengths, decode lengths, and agent execution patterns, the observed slowdown is consistent with our profiling results, further demonstrating the robustness of SM partitioning in dynamic MAS workflows. Some agents exhibit particularly low slowdown because no eligible proactive prefill workload is available during their decoding.
2. Memory Overhead
We have provided detailed measurements of peak GPU memory usage and memory overhead in our common response, “Memory Overhead and Performance under Memory Constraints.”
Regarding the suggestion to vary context length and the number of pending agents to approach device capacity, directly controlling these characteristics in real-world MAS workflows is challenging due to uncontrollable agent behavior. Instead, we vary the GPU memory budget available to the workflow to simulate memory-constrained conditions approaching device capacity. Detailed results are presented in the “Performance under Memory Constraints” section of our common response.
Regarding scalability, MessageFlow inherently involves a memory–performance tradeoff. As context lengths and the number of pending agents increase, more KV states may need to be retained. However, MessageFlow enforces a configurable GPU memory budget, preventing unbounded memory growth. With a larger memory budget, more proactive KV states can be retained and reused, potentially enabling greater acceleration.
3. Adaptation to Message Updates
First, MessageFlow adopts a relatively conservative strategy: it performs proactive prefill only on prefixes identified as stable, rather than speculatively computing potentially unstable content. Therefore, under the supported message semantics, previously computed prefixes are not invalidated by subsequent valid message updates and can all be reused when the corresponding agent is invoked.
MessageFlow already supports OVERWRITE semantics, as described in Section 3.1. Specifically, an overwritten message becomes eligible for proactive prefill only after it is guaranteed that no agent can further modify it. We consider message reordering and summarization as forms of OVERWRITE, since they effectively replace existing message content.
We acknowledge that frequent overwriting, reordering, or summarization can delay prefix stabilization, reducing precomputation coverage and available overlap opportunities. However, such updates do not introduce wasted proactive computation, since MessageFlow defers prefill until the corresponding prefix becomes stable. Consequently, MessageFlow would approach conventional execution when few stable prefixes are available, rather than incurring substantial overhead from invalidated precomputation.