- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents ACCORDION, a runtime optimization for multi-agent LLM workflows with parent-child dependencies. Instead of waiting for a complete parent message, a child starts generating from partial context and checks its draft when the parent supplies more information. The two main design choices are a surprisal-based rule for sending context updates and prefix verification with rollback to retain useful child computation. The paper also analyzes the relationship between information in a context update and prefix reuse. Experiments on reasoning and code-generation tasks, several dependency graphs, and a dynamic plotting workflow report up to 46.6% lower latency than sequential execution. ACCORDION also provides additional acceleration when combined with EAGLE-3.
Critical strengths
-
Overlapping parent and child decoding is a useful idea. Batching and prefix caching alone cannot eliminate the wait for a complete parent message. Verification and rollback give a concrete way to retain useful child computation as the context changes.
-
The evaluation covers different graph structures and model sizes, streaming-prefill overlap, and composition with EAGLE-3. I particularly appreciate the OpenManus experiment, where tools and revision rounds change the workflow during execution. The discussion of synchronization and cache-recovery costs also helps explain where the method is useful.
Critical weaknesses
-
The practical benefit depends on spare serving capacity. Speculation creates additional verification and discarded computation, which can become costly when many workflows share the GPUs. Appendix E.4 examines this issue, but uses fixed 16-token commits and controlled acceptance rates. The operating range of the default policy on actual task workloads therefore remains unclear.
-
The evidence for the adaptive commit rule is less strong than the evidence for early child execution. Fixed-size commits recover much of the latency benefit and are sometimes equally good or faster. For example, Chunk-16 and ACCORDION have the same accuracy and almost the same latency on six-agent HumanEval. The adaptive rule may reduce the need to choose different chunk sizes for different tasks, but this advantage deserves a clearer explanation.
Questions for authors
-
How do latency and throughput change under increasing concurrent load on the benchmark tasks, using the default ACCORDION policy? At what load does the extra speculative work outweigh the benefit of overlapping generation?
-
How does the surprisal rule compare with one fixed chunk size chosen on validation data? Does it give a more consistent latency-quality trade-off across tasks, or mainly reduce the need for task-specific tuning?
-
Does the same commit threshold work when parent and child agents use different model families or have very different capabilities? The OpenManus result is a useful heterogeneous example. Are there results for other mixed parent-child model pairs?
4 comments
Reviewer GWP7 (2/2)
W2 & Q2: the evidence for the adaptive commit rule, and the rule vs. one fixed chunk chosen on validation data.
It achieves both: with and adapted online across requests, our surprisal rule is more consistent across tasks than a validation-selected fixed chunk and eliminates task-specific tuning altogether. Early child execution with verification and rollback is the foundation every streaming policy in Table 1 shares; the commit rule decides when children receive new parent text, and therefore how well that foundation is used. Because optimal granularity depends on the task, ACCORDION-online in Section 3.4 updates and online between requests from completed parent surprisal streams, needing no validation set.
For the direct comparison you requested, we selected the fastest single chunk size on 200 held-out questions across four- and six-agent MMLU and GSM8K in Appendix F.6, which chose Chunk-64:
Table R3. Speedup in Table 1 averaged over three runs, for 3 / 4 / 5 / 6 agents.
| Policy | MMLU | GSM8K | HumanEval |
|---|---|---|---|
| Chunk-64, validation-selected | 1.23 / 1.49 / 1.61 / 1.54 | 1.41 / 1.54 / 1.50 / 1.57 | 1.11 / 1.34 / 1.20 / 1.40 |
| ACCORDION-online | 1.20 / 1.46 / 1.55 / 1.55 | 1.38 / 1.50 / 1.49 / 1.51 | 1.29 / 1.61 / 1.59 / 1.79 |
Across all 12 settings, ACCORDION-online achieves the highest overall mean speedup at 1.49×, beating both the best single chunk in hindsight, Chunk-48 at 1.43×, and validation-selected Chunk-64 at 1.41×. More importantly, it stays within 4.0% of the best fixed chunk size in every setting, whereas every fixed chunk falls 15–44% behind in at least one, with Chunk-64 trailing by up to 26% on HumanEval in Appendix F.6.
Addressing your three sub-questions:
- A more consistent latency trade-off? Yes, because the cost of a commit is task-dependent and ACCORDION-online measures it directly, as analyzed in Appendix F.4. On MMLU, one commit rebuilds 2.3–3.4 running children and discards 24–47 drafted tokens per rebuild, so sparser commits win and it selects around 28–30. On HumanEval, one commit rebuilds only 1.2–1.5 children and discards under one token, so frequent commits win and it selects around 1.75–3.5. Because larger deliveries discard more drafts, stays at most 64.
- A more consistent quality? Accuracy is reliably preserved: ACCORDION-online's mean accuracy change across the 12 settings is −0.11 pp, and every schedule in Table 1 remains within run-to-run variation.
- Mainly less task-specific tuning? Yes, it removes per-task tuning entirely. Its warm-start point, and , was chosen once during development on the 200 four-agent MMLU questions of Table 1, generalizes at 1.42–1.45× on unseen four-agent MMLU sets, and is used unchanged for every other task, model, and topology. Because code generation favors frequent commits whereas MMLU and GSM8K favor coarser ones in Appendix F.3, every fixed chunk size lags on some workload. ACCORDION-online avoids manual sweeps altogether, pricing every candidate between requests in 21–39 ms and settling to adjacent grid values in the second half of each run, as shown in Appendix F.5.
In the revised manuscript, Section 3.4 and Appendix F present ACCORDION-online, the introduction notes that and are adapted online across requests, and the abstract, contributions, and Section 5.2 highlight that ACCORDION-online achieves the highest mean speedup in Table 1 while staying within 4.0% of the best fixed chunk in every setting.
These results answer all three questions with direct measurements: where ACCORDION pays off under load, how its online commit rule outperforms any single fixed chunk across tasks, and why one threshold carries across model families. We would be grateful if you would reflect this in your score, and we are happy to provide any further detail.
Thank you for providing the detailed response to my questions. I think this addresses most of my concerns. Good luck!
Thank you for the quick reply! We're glad the response addressed most of your concerns, and your review genuinely made the paper better.
If anything still holds you back, just let us know and we'll address it before the discussion closes. Otherwise, if you feel the revision has resolved your concerns, we'd be grateful if you could consider raising your score.
Reviewer GWP7 (1/2)
Thank you for your positive assessment and precise systems questions. We are glad you found verification and rollback a concrete way to preserve useful child work and valued the dynamic OpenManus experiment. We answer each question below with new measurements.
W1 & Q1: behavior under concurrent load, and where speculative work outweighs the benefit.
Across real four-agent workflows on one engine, ACCORDION delivers substantial throughput and latency gains throughout the low-to-moderate load regime, raising throughput by 1.43–1.47× and cutting P50 latency by 30–34% at c=1, before reaching break-even at about 4 workflows in flight on MMLU and between 4 and 8 on GSM8K. Moving beyond the synthetic study in our original Appendix E.4, this closed-loop sweep runs the default policy on four-agent MMLU and GSM8K with c workflows in flight on one engine:
Table R1. Workflow throughput of ACCORDION relative to Sequential.
P50 / P95 workflow latency including queueing, Sequential → ACCORDION:
Accuracy shows no trend with load, staying within ±4 pp across concurrency levels. Past the crossover, ACCORDION's extra work from prompt rebuilds, verification, and discarded drafts saturates the engine. Sequential, which runs only one agent per workflow at a time, is far less affected: from c=1 to c=8 its P50 latency rises 12–22%, whereas ACCORDION's roughly doubles. MMLU crosses over earlier because its parents revise more heavily, discarding 30–41% of child tokens versus 8–16% on GSM8K in Table R7.
Because the server directly tracks how many workflows are in flight, a calibrated admission threshold that falls back to Sequential above the crossover is never worse than Sequential across this sweep while retaining 1.14–1.47× at low load. SmartSpec by Liu et al. (2024) applies the same principle to model-level speculative decoding. The revision adds this sweep with P50 and P95 latencies to Appendix E.4 and deployment guidance to Appendix G.
Q3: the same threshold for parent and child models of different families or capabilities.
Yes: with τ=10 and Mmax=64 unchanged, mixing Qwen2.5-14B-Instruct and Llama-3.1-8B-Instruct across model families and sizes yields 1.33–1.58× speedups over Sequential in both directions while preserving accuracy. On four-agent MMLU and GSM8K, each model in the mixed runs occupies its own engine and A100 GPU; commits cross engines as plain text, each child tokenizes updates with its own tokenizer, and Sequential uses the exact same placement.
Table R2. Cross-family runs: speedup with paired 95% CI, and accuracy from Sequential → ACCORDION.
The single-family reference rows run on one GPU, so each row should be compared only with its own Sequential baseline; every accuracy change has a paired 95% CI that includes zero. Crucially, the mechanism is agnostic to model pairing: the commit signal uses only the parent's own token NLL, and retention is decided only by the child's own Top-5 check, so neither assumes a shared tokenizer or model family.
While the best τ naturally scales with the parent's NLL, ACCORDION-online in Section 3.4 selects τ and Mmax directly from the parents' own surprisal streams, so a new model pair requires no sweep. OpenManus in Appendix E.3 provides an additional multimodal heterogeneous case with Qwen2.5-VL-7B and Qwen2.5-14B, reaching 1.18× at τ=10. Table R2 is added as Appendix E.10.