acceptodds
Under review as a conference paper at ICLR 2027

Dynamic Workflow Routing for Multi-Agent Systems

Abstract

We introduce a training-free framework for adapting workflows in large language model multi-agent systems at inference time. Rather than committing every request to a predetermined sequence of planning, generation, and refinement, this selects subsequent actions based on the outputs produced during execution. This flexibility addresses a limitation of fixed workflows: routine requests can incur unnecessary computation, while complex or ambiguous requests may receive inadequate reasoning and verification. Routing follows a two-tier design that balances decision cost with oversight. A lightweight Scout checks progress at each stage, escalating ambiguous or high-risk cases to a more capable LLM-based Router. These assessments determine whether the workflow should terminate, proceed, or introduce corrective steps, allowing computational effort to respond to the needs of each instance. Across the evaluated benchmarks, this achieves reductions of 46.1% in token consumption and 47.8% in end-to-end latency, demonstrating the efficiency benefits of adapting agent coordination during execution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multi-agent system, LLM agent system, agentic system, system efficiency, Multi-agent workflow, Workflow routing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RjtiOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DynaMAS controls multi-agent workflow execution using intermediate outputs. A lightweight Scout combines deterministic checks and a small LLM to label each checkpoint output CONTINUE, EARLYEXIT, or AUGMENT, a stronger Judge confirms proposed exits or selects corrective workflows, and a background Summarizer updates a failure-type mapping and synthesizes new workflows from completed traces. Across four benchmarks, the paper reports average reductions of 46.1% in output tokens and 47.8% in latency relative to AFlow, with preserved code and math accuracy and improved HotpotQA F1.

Critical strengths
  1. Mid-execution routing is a well-motivated idea. The paper shows that a static workflow over-provisions easy queries and under-provisions hard ones (Section 3), and its insight that the need for more computation often appears only mid-workflow sets it apart from routers that decide once per query. Deciding at checkpoints from intermediate outputs, with a lightweight Scout that escalates to a stronger Judge, lets one system both exit early and add corrective workflows, and it layers onto an existing searched workflow, calibrating its thresholds on validation data instead of training a separate router.

  2. The design is supported by controlled evidence. Ablations show that, on HotpotQA, the two-tier cascade nearly matches judging every checkpoint at a fraction of the latency, and that removing exit verification lowers task performance on every benchmark; the token and latency savings over AFlow are also large and far exceed run-to-run variation while code and math accuracy stay at AFlow's level. This makes the efficiency claim credible, subject to the issues below.

Critical weaknesses
  1. The experiments do not yet show that the efficiency comes without loss. The headline claim is that DynaMAS cuts tokens and latency while keeping accuracy, and two pieces of evidence for it are missing. First, exit reliability is not measured at the deployed threshold. The Scout's best MBPP operating point has 65.7% precision (Section 5.4), while the Judge confirms 90.1% of MBPP exit proposals (Table 8); if both figures describe the same proposals, more than a quarter of confirmed exits would be incorrect, which would need to be reconciled with MBPP accuracy matching AFlow. Second, it is unclear whether the system learns from test labels. Appendix C freezes calibration on validation data before testing, but Section 4.2 says the failure mapping and candidate set adapt continuously as queries arrive, and Algorithm 2 learns from verified outcomes that Appendix E.3.1 defines using gold answers and failing unit tests. Measuring exit precision at the deployed threshold, and rerunning with all state frozen after validation, would settle both points.

  2. The reported savings may not include the full controller cost. Table 2 says the Judge is invoked at 15.4% of checkpoints, but the Judge is called on every EARLYEXIT or AUGMENT decision (Section 4), and Table 8's 35.5% EARLYEXIT plus 24.1% AUGMENT on MBPP implies Judge calls at 59.6% of checkpoints. Table 2 likewise lists the Scout at 45.7%, although the Scout runs at every checkpoint. Since each Judge call costs about as many tokens as a workflow agent (Table 2), and the paper does not say which Scout, Judge, Summarizer, and warm-up costs Tables 1 and 7 include, the 46.1% output-token and 47.8% latency reductions cannot yet be read as full inference-cost savings.

Questions for authors
  1. Evaluation protocol: Which structures update during held-out testing, do those updates use the verified outcomes defined in Appendix E.3.1, and when do those updates occur relative to warm-up?
  2. Tree and accounting: What counter and initialization control splitting, how is the failure posterior normalized, and does the mapping, which gives every checkpoint the final-run label, predict current corrective needs or eventual run failure? How do Table 2's rates reconcile with Algorithm 1 and Table 8, which Scout, Judge, Summarizer, and warm-up costs do Tables 1 and 7 include, and how are failed selections counted in routing quality?
  3. Frozen state and exit reliability: Do the results hold with all state frozen after validation? At the deployed thresholds, what is exit precision, how often are confirmed exits incorrect, and how often does DynaMAS fail where the original workflow succeeds?
  4. Attribution: The Judge has adaptive access to seven code and eight RAG seed workflows plus synthesized ones, and the Judge-only variant already reaches 43.94 F1 on HotpotQA against the always-execute baseline's 38.72 F1 (Table 4). How does AFlow perform with the same workflow pool and matched controller access?
  5. Positioning: Section 3 states that prior methods commit to a single fixed workflow for all queries, yet MaAS and MasRouter, both compared in Table 1, already build query-dependent systems. The Scout-then-Judge escalation also follows LLM cascades such as AutoMix (Aggarwal et al., 2024, "AutoMix: Automatically Mixing Language Models") and the cascaded judges of Trust or Escalate (Jung et al., 2025, "Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement"), which Section 2.2's brief contrast with cascades does not address, and MASC (Shen et al., 2026, "Metacognitive Self-Correction for Multi-Agent System via Prototype-Guided Next-Execution Reconstruction") detects and corrects step-level errors during execution. How does DynaMAS differ from these works, and is its specific addition per-checkpoint three-way routing with an online-grown workflow library?
5 comments
AuthorsOpenReview ↗

response to Reviewer Rjti (part 1)

Response to Reviewer Rjti

Thank you for the very detailed review. We will answer your questions and provide additional evaludation to support our claim.

W1a / Q3. Exit reliability, and 65.7% vs. 90.1%.

The two figures describe different sets of exit proposals. The 65.7% comes from the diagnostic sweep in Sec. 5.4, not from the deployed exit rule, and the two differ in three ways:

  • Test cases. The sweeping is done on all agents on all workflow on the training set. Thus the base accuracy of all agents' output is around 60% much lower than the outputs during execution which is around 90%.
  • Threshold. The deployed τ_good (0.80 on MBPP; Table 5) is stricter than the sweep's high-recall point: it proposes about 47% of the possible exits, not 95%.
  • Rule. Deployment also requires φ_format = 1 and Judge confirmation. The 90.1% is the Judge's confirmation rate, not a precision.

Measured at the deployed thresholds:

At deployed thresholdsMBPPHotpotQA
Scout exit precision / recall (before the Judge)88% / 47%90% / 78%
Precision of confirmed exits96%94% (exact match: 86%)
Wrong where AFlow is right / right where AFlow is wrong1.5% / 2.0% (net +0.5, Table 1)2.5% / 7.3% (net +4.8)

This confirms that early_exit doesn't affect the correctness. 96% of confirmed MBPP exits are correct, and an exit loses an answer that AFlow gets right on only 1.5% of queries, outweighed by 2.0% the other way. On HumanEval and GSM8K, confirmed-exit precision is 96.5% and 96%, with regressions vs. rescues of 1.2% vs. 1.3% and 1.0% vs. 1.4% (Table 1's +0.1 and +0.4). The HotpotQA net counts F1 ≥ 0.5 as correct; under exact match it is still +4.1. We will add the deployed operating point to Sec. 5.4 and Fig. 6.

The savings are not just truncation. On CONTINUE and AUGMENT, DynaMAS runs one refine step or one short corrective workflow instead of AFlow's full remaining pipeline. But with routing disabled (first agent, one refine, then stop), accuracy falls to 84.0 ± 0.4 on MBPP and 35.5 ± 0.6 F1 on HotpotQA, about Self-Refine's level (Table 1). Removing exit verification costs 2.9 points on average for only a 5.8% token saving (Table 9).

W1b / Q1 / Q3. Does the system learn from test labels?

No. Gold outcomes (App. E.3.1) are used only during the validation warm-up, which seeds the failure-type tree and A_syn; this is the phase shown in Fig. 8. In the reported runs, the tree, pool, and thresholds are then frozen, and Algorithm 2 (including its reliability decay) does not run at test time. The only state that changes at test time is the φ_hist reliability EMA, which is driven by Judge confirmations and the deterministic checkers, never by gold labels. We will fix the "adapts continuously" wording in Sec. 4.2. We also reran all four benchmarks under three stricter settings:

MBPPHumanEvalGSM8KHotpotQA
Paper (Table 6)94.7 ± 0.2896.0 ± 0.2796.3 ± 0.3343.6 ± 0.57
Fully frozen (φ_hist frozen too)94.6 ± 0.395.9 ± 0.396.2 ± 0.343.3 ± 0.6
Label-free warm-up (no gold labels anywhere)94.3 ± 0.395.8 ± 0.396.0 ± 0.442.8 ± 0.6
Label-free, with pool and tree updating online at test time (deployment mode)94.4 ± 0.395.9 ± 0.396.1 ± 0.343.1 ± 0.6

The label-free warm-up replaces gold outcomes with executor-written tests (code), self-consistency (math), and the Judge's evidence-entailment check (RAG). It never uses pass/fail results on the benchmark's own asserts. At inference time, all methods, AFlow included, see the same prompt-visible tests (AFlow's protocol; App. G). The fully frozen run is within one standard deviation of Table 6, and every change is at most 0.8 points (at most 1.5 standard deviations).

Data splits. MBPP and GSM8K use their official test sets (500 and 1,319 problems). HotpotQA's test labels are hidden, so "test" is the 7,405-question full-wiki dev set. HumanEval has no train or validation split, so we hold out 33 problems for calibration and test on the remaining 131. We will state all of this in Sec. 5.1.

AuthorsOpenReview ↗

response to Reviewer Rjti (part 2)

W2 / Q2. Table 2 rates and cost accounting.

Thank you for checking this so carefully, and we apologize for the misleading caption. The invocation rates in Table 2 are relative to all agent calls of the baseline AFlow workflow on the same queries. The Scout does run at every DynaMAS checkpoint, but DynaMAS reaches far fewer checkpoints than AFlow makes agent calls, because it exits early or replaces the remaining pipeline with one short corrective workflow. That is why the Scout's rate is 45.7% rather than 100%. We will correct the misleading caption.

On cost, the reductions in Tables 1 and 7 are full inference-cost savings. Both tables include every call on the inference path: workflow agents, Scout, and Judge (including the Judge's reasoning tokens). Table 2 gives each unit's tokens per call, and Table 7 (App. C.2) gives the complete input, output, and total accounting. Per MBPP query, DynaMAS's cost breaks down by unit as follows:

MBPP, tokens per queryInputOutputTotal
Workflow agents8171,5332,350
Scout34392435
Judge6128051,417
DynaMAS, inference (sum = Table 7)1,7722,4304,202
DynaMAS, amortized over 1,000 queries4,802 (4,202 inference + 600 warm-up)
AFlow (Table 7)3,8766,64310,519

The summarizer and the pilot run tokens are for one-time only. And these token consumptions can be amortized over the future runs. Even adding this additional overhead, our framework still has substantial token consumption reduction compared to baselines.

  • Summarizer and warm-up. All Summarizer calls happen during warm-up, so we report them, together with the pilot runs and the warm-up stream, as a one-time cost of about 600K tokens on MBPP. Over 1,000 queries, this adds only around 600 tokens per query (the amortized row above).

Q2. Tree details.

  • Counter and splitting. Each node keeps decayed counts n_v ← (1 − η_M) n_v + e_τ, with N_v = Σ_τ n_v,τ. App. A mistakenly prints η_M on the one-hot term; as printed, N_v could never exceed 1, so the N_v ≥ 50 split rule could never fire. We will fix this typo. With η_M = 0.01, N_v saturates at 100 and passes 50 after about 69 observations. The node then splits if the best threshold's KL gain exceeds 0.1, and a child is pruned when its KL divergence from its parent stays below 0.02 over a sliding window. The tree is initialized from pilot labels plus a Dirichlet prior α.
  • Normalization. Each node term (α_τ + n_v,τ) / (α_0 + N_v), with α_0 = Σ_τ α_τ, is a distribution over failure types. We set λ_0 = 1 at the root and λ_j = N_vj / (N_vj + κ) below it, so the weights ω_j sum to 1.
  • What the tree predicts. Each checkpoint is labeled with the run's final failure type, so the tree predicts "what this run will eventually fail on, given what we see now." This is the prior the Judge needs. Updating only from the Summarizer's evidence_step onward (App. E.3.1) gives +1.5 Top-1 on HotpotQA (74.7 → 76.2) with the same F1.
  • Failed selections count as misses for Top-1 and Top-2 in Table 3, and as Q = 0 when computing Q_t. With this convention, the mean Q_t on HotpotQA is 0.88.
AuthorsOpenReview ↗

response to Reviewer Rjti (part 3)

Q4. Attribution. We have done additional evaluation to show that even if the baselines were given the same additional workflow pool, our routing design can still have higher accuracy and lower token consumption. We gave the baselines the same final workflow pool, and gave AFlow the same Judge and tree prior as well:

MethodHotpotQA F1 / total tokensMBPP Acc. / total tokens
AFlow38.7 / 18,27394.2 / 10,519
AFlow + the most-used pool workflow, always appended40.5 ± 0.6 / 22,70094.4 ± 0.3 / 15,400
AFlow + same Judge, pool, and tree prior, choosing a fix after the final step41.5 ± 0.6 / 21,60094.6 ± 0.3 / 14,000
MasRouter + same pool (A0 only, Table 10: 32.2 / 89.8)37.5 ± 0.7 / 17,50093.0 ± 0.5 / 8,500
DynaMAS43.6 / 12,77394.7 / 4,202

In the post-hoc variant (third row), the Scout runs once on AFlow's final output, so the Judge sees the same features and tree prior as in DynaMAS. DynaMAS still leads it by 2.1 F1 on HotpotQA (paired bootstrap, 95% CI +1.5 to +2.7) with about 40% fewer tokens. On MBPP, the AFlow variants tie DynaMAS on accuracy (MasRouter + pool is 1.7 points lower), and DynaMAS is 2–4× cheaper than all three. Deciding mid-run matters because the fix lands where the failure first shows up (for example, a missing second hop is caught before the answer is written), and easy queries exit early. To isolate this with identical components, we also restricted the same controller to a single decision before execution: it then reaches 40.0 F1 on HotpotQA, vs. 43.6 when it decides at every checkpoint.

On Table 4, Judge-only is the "Judge at every checkpoint" ceiling. The cascade reaches 99.3% of its F1 (43.63 vs. 43.94) with 71% lower latency (125.4 s vs. 437.5 s) and 31% fewer tokens.

Q5. Positioning.

  • MaAS and MasRouter are query-dependent, but they decide before execution. We will correct Sec. 3 accordingly.
  • AutoMix and Trust or Escalate escalate between models to produce one answer or one judgment. We adopt their cascade principle (and, like Trust or Escalate, conformal thresholds), but DynaMAS routes workflow actions inside a multi-agent trace. We will cite and credit both.
  • MASC corrects anomalous steps with a single correction agent, with reliability as its goal. It has no cost-driven exit, and its correction action is fixed.
  • Our specific addition is as you describe it: per-checkpoint exit / continue / augment routing, with a Judge-verified exit and an online-grown, failure-typed workflow library, training-free and layered on an existing workflow. As a direct check, an AutoMix-style self-verification exit reaches 93.0 at about 3,600 output tokens on MBPP and 38.0 F1 at about 10,500 on HotpotQA, vs. 94.7 at 2,430 and 43.6 at 8,648 for DynaMAS.

Thank you again. In the revision, we will correct the misleading words and add additional evaluations to the paper.

Reviewer_EPqoOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes what the authors claim to be the first dynamic workflow routing framework for LLM-based multi-agent systems (MAS).

Critical strengths

The proposed idea is reasonable and addresses a relevant research problem.

The paper is generally understandable and well structured.

Critical weaknesses
  • Overstated claims: Some claims regarding the novelty of the proposed approach and the limitations of existing methods appear to be overstated.

  • Insufficient technical details: Some critical aspects of the proposed framework are not sufficiently specified, making it difficult to fully assess its technical contributions.

Details refer to the Questions section.

Questions for authors
  1. Motivation and novelty: The motivation of this study is largely based on the observation that existing multi-agent systems rely on static workflows. However, Sections 2.1 and 2.2 discuss several recent studies involving automatic workflow construction (Lines 107–108) or adaptive workflow allocation (Lines 126–127). This appears inconsistent with the paper's central motivation and its claim to be the first dynamic workflow routing framework. Could the authors clarify whether dynamic workflow approaches already exist and, if so, what distinguishes the proposed framework from them? If not, how should the term "dynamic" be understood, given that the existing approaches discussed in Sections 2.1 and 2.2 appear to address similar concepts?

  2. Workflow pool construction (Figure 3): How is the workflow pool in Figure 3 determined and constructed? Could the authors clarify the criteria for selecting workflows and whether the pool is predefined, automatically generated, or dynamically updated during execution?

  3. Comparison with existing automatic workflow methods: What are the key differences between the proposed framework and a simpler approach that combines an existing automatic workflow method with workflow pool-based initialisation? Could the authors clarify what additional technical contributions or capabilities the proposed framework offers beyond such a combination?

5 comments
AuthorsOpenReview ↗

Response to Reviewer EPqo(part 1)

Response to Reviewer EPqo

Thank you for the review and your careful reading of this paper.

Q1. What does "dynamic" mean, given the prior work in Sections 2.1 and 2.2?

Thanks for raising this. The paper doesn't claim to be the first dynamic workflow framework. In fact, our main advantage is training-free. Previous methods require a dedicated time-consuming training process so that their method can 'dynamically predict' the best workflow for a query. We don't require this long training process. With a short warm-up phase, our can choose the best workflow for each query based on the intermediate state during running

What sets DynaMAS apart is the combination of two properties:

  1. It is training-free. Learned routers have to train a routing policy for each task: MaAS and MasRouter at the query level, and Puppeteer (Dang et al., 2026) step by step with RL. Search-based methods (AFlow, ADAS) run a costly offline search and then reuse a single workflow. DynaMAS needs only a short warm-up and threshold calibration on validation data (about 600K API tokens on MBPP; see our response to jbLa, Q2), and it sits on top of any existing workflow.
  2. It routes on the intermediate state, not just the input. Query-level routers (MaAS, MasRouter) decide once, from the input, before anything runs (Sec. 1, lines 064–067). DynaMAS decides at each checkpoint, from the current intermediate output, whether to stop, continue, or add a corrective workflow. Some prior methods also act mid-run: Flow re-plans the workflow graph, Puppeteer picks the next agent, and MASC corrects step-level errors. None of them, however, combines a verified, cost-driven early exit with failure-typed fixes drawn from a library that grows over time, and does so without training.

So in our paper, "dynamic" means that the computation each query receives is decided during execution, from its intermediate outputs, rather than fixed for the whole task or chosen once from the input. To isolate this effect with identical components, we restricted the same controller to a single decision before execution. On HotpotQA it then reaches 40.0 F1, vs. 43.6 when it decides at every checkpoint.

Q2. How is the workflow pool (Figure 3) built?

The pool is A_t = A0 ∪ A_syn (Sec. 4.2.2): a predefined library of hand-written corrective workflows (A0), plus workflows that the Summarizer adds automatically during a short warm-up (A_syn).

  • A0 (predefined): 7 code and 8 RAG corrective workflows (App. G, Tables 14–15), such as self-consistency voting, test-driven repair, second-hop retrieval, and NLI-based hallucination checking. Each one targets specific seed failure modes (Tables 12–13). Pilot runs on validation data keep the workflows that fix at least one pilot failure. Nothing is tuned on the test set.
  • A_syn (generated automatically): during the validation warm-up, the Summarizer proposes a new workflow only for a diagnosed failure that no existing workflow covers. A proposal is admitted only if it parses, is not a duplicate, and passes a dry run. On average, 7.8 workflows are proposed, 7.0 are admitted, and 6.4 are actually used (Table 11), and the set stops growing after about 60 queries (Fig. 8).
  • Is the pool updated during execution? Not in the reported benchmark runs. There, the pool, the thresholds, and the failure-type tree are frozen so that the evaluation is clean. App. C already states this for calibration, pilot seeding, workflow admission, and thresholds, and we will add the tree to that sentence. In deployment mode, the pool and tree keep updating during execution from label-free signals. That mode reaches 43.1 F1 on HotpotQA (see the table in our response to Reviewer Rjti), vs. 43.6 in the paper's run. This is what "online-adaptive" in the title refers to.
AuthorsOpenReview ↗

Response to Reviewer EPqo(part 2)

Q3. Why not simply combine an existing workflow-search method with pool-based initialization?

Thanks for asking this question. That combination still produces a static workflow. Adding the pool's workflows to AFlow's search enlarges the search space, but the search still returns one workflow per task, for example a single workflow for every coding query. As Sec. 3 shows, different queries need very different amounts of work.

Choosing per query is hard because both kinds of mistake are costly. A workflow that is too simple loses accuracy (IO and the small workflows in our response to jbLa, W1), and one that is too heavy wastes tokens (AFlow). DynaMAS handles this with its two-tier structure: a cheap Scout checks every intermediate output, and the Judge is called only on uncertain cases, to confirm an exit or pick a fix. To our knowledge, no prior training-free method decides from the intermediate result whether a running query should stop early or escalate to a heavier corrective workflow.

We tested the combination you suggest directly:

MethodMBPP Acc. / total tokensHotpotQA F1 / total tokens
AFlow (paper)94.2 / 10,51938.7 / 18,273
AFlow, searched with A0 added as operators94.5 / 11,50040.0 / 21,000
DynaMAS on top of that workflow94.8 / 5,90044.0 / 14,700
Puppeteer (same agents and executor), trained on our validation split92.5 / 6,80038.0 / 15,500
Puppeteer, trained on all training data (MBPP: 374 problems; HotpotQA: 5k)93.0 / 6,70040.5 / 15,000

A larger search space makes the static workflow slightly better but more expensive, and DynaMAS improves whichever workflow it sits on. The before-vs.-during comparison in Q1 isolates the same effect with identical components. Puppeteer, the closest step-wise method, learns when to stop but cannot add workflows outside its agent pool, so it stays 3.1 F1 behind DynaMAS even with full training data. It also needs about 9M rollout tokens plus GPU training per task, vs. DynaMAS's one-time cost of about 600K API tokens on MBPP, with no training.

If you have other questions, please let us know.

Thank you for clarifying the motivation. I am still trying to understand the advantages of the training-free approach discussed in Lines 251–258. However, I do not find the current justification sufficiently convincing. In particular, the discussion does not acknowledge any potential limitations or trade-offs of adopting a training-free approach, making the claimed advantages appear overly optimistic. Could the authors point me to any discussion or experimental evidence in the original submission that provides a more balanced assessment of these claims?

Reviewer_jbLaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that static multi-agent workflows found by AFlow/ADAS apply the same steps to every query, over-provisioning easy queries and under-provisioning hard ones. DynaMAS adds a two-tier cascade router at each step of an existing workflow: a Scout (rule-based checkers plus a Llama 3 8B rubric scorer) classifies each intermediate output as CONTINUE, EARLY EXIT, or AUGMENT; a Judge (Doubao-seed-1.6-lite) confirms early exits and, on AUGMENT, selects a corrective workflow from a candidate set. An asynchronous Summarizer (Doubao-seed-2.0-pro) maintains an online failure-type decision tree and synthesizes new candidate workflows. On MBPP, HumanEval, GSM8K, and HotpotQA, applied on top of the AFlow pipeline, it reports 46.1% fewer output tokens and 47.8% lower latency on average, with accuracy preserved or slightly improved.

Critical strengths
  1. Well-motivated and practical. Figure 2 quantifies over-provisioning convincingly (on MBPP and HumanEval, over 80% of instances are solved by the first agent and fewer than 6% need the full workflow), and the router is a training-free policy layer that can be placed on top of an existing pipeline.
Critical weaknesses
  1. The savings mostly reflect an over-provisioned baseline on saturated benchmarks. AFlow already reaches 94–96% on MBPP, HumanEval, and GSM8K, and the paper's own Figure 2 shows most MBPP/HumanEval queries need only the first agent, so any reasonable early-exit rule would cut tokens substantially. Relative to single-call IO, DynaMAS uses about 4× the output tokens on HumanEval (5,877 vs. 1,425) for +3.4 points and about 6× on GSM8K (8,624 vs. 1,343) for +4.6 points; measuring savings only against the most expensive pipeline overstates the benefit of routing. In addition, Table 1 reports output tokens only; the Judge reads a compressed trace, features, and the candidate set, and the Summarizer reads full traces and writes workflow code, yet it is not stated whether Table 1's token counts include Scout, Judge, or Summarizer outputs, and the Summarizer's per-query cost before saturation and the pilot-run cost are not reported.
  2. The accuracy gain is attributable to stronger models rather than to routing. The only substantive accuracy gain is on HotpotQA (38.7 → 43.6 F1). In Table 4, Scout-only gives 37.24 F1, below the 38.72 baseline, while Judge-only (Judge at every checkpoint) gives 43.94, above the full method's 43.63; the gain thus comes from invoking the Judge. Appendix C.5 further shows that the first two synthesized workflows alone raise HotpotQA F1 from 36.1 to 42.0 (and MBPP from 86.3% to 92.1%), i.e., much of the gain comes from workflows written by Doubao-seed-2.0-pro. The executor model used by all methods is never specified, so if the Judge and Summarizer are stronger than the executor, the comparison with baselines is not on equal terms.
Questions for authors
  1. (Clarification) Which executor model do all methods use? Are the Judge and Summarizer stronger than the executor?
  2. (Clarification) Do Table 1's token counts include Scout, Judge, and Summarizer outputs? What is the total cost including input tokens, Summarizer synthesis, and pilot runs?
  3. (Clarification + additional experiment) In what setting were the Appendix C.5 synthesis gains (MBPP +5.8, HotpotQA +5.9) measured? How much of DynaMAS's advantage remains if baselines can also use these synthesized workflows?
  4. (Suggestion) Does accuracy hold on harder, non-saturated benchmarks?
5 comments
AuthorsOpenReview ↗

response to Reviewer jbLa (part 1)

Response to Reviewer jbLa

Thank you for the careful and constructive review. Your questions about the executor, the baselines, and the full cost are the right ones to ask. We answer each one below, with new experiments where needed.

Q1. Which executor do all methods use? Are the Judge and Summarizer stronger than the executor?

All methods use the same executor, Doubao-seed-1.6-lite. This includes every baseline and the AFlow pipeline that DynaMAS runs on. The Judge is the same model as the executor, so it adds no model capacity. Where the paper calls the Judge "stronger", it means stronger than the Scout, and we will say so explicitly. The Scout (Llama 3 8B) is weaker than the executor. Only the Summarizer (Doubao-seed-2.0-pro) is stronger, and in the reported runs it is used only during warm-up.

To rule out "a stronger model did it", we reran DynaMAS with every role set to the executor model:

HotpotQA F1MBPP Acc.Total tokens (HotpotQA / MBPP)
AFlow (Tables 1, 7)38.794.218,273 / 10,519
DynaMAS (paper)43.694.712,773 / 4,202
DynaMAS, all roles = executor42.6 ± 0.694.4 ± 0.312,900 / 4,250

Without any stronger model, DynaMAS keeps 3.9 of its 4.9 F1 points on HotpotQA, at essentially the same cost.

Q2. What do the token counts include, and what is the full cost?

Tables 1 and 7 count every model call on the inference path: the workflow agents, the Scout, and the Judge (including the Judge's reasoning tokens). Table 1 reports output tokens, and Table 7 reports input, output, and total. The Summarizer, the pilot runs, and the warm-up stream are a one-time cost paid only during warm-up: about 600K tokens on MBPP. Amortized over 1,000 queries, this adds only 600 tokens per query, and the share keeps shrinking as more queries are served.

MBPP, per queryCallsInputOutputTotal
Workflow agents1.98171,5332,350
Scout1.234392435
Judge0.726128051,417
DynaMAS, amortized over 1,000 queries4,802 (4,202 inference + 600 warm-up)
AFlow≈5.4 agent calls3,8766,64310,519 (Table 7)

Even with the warm-up included, DynaMAS uses 54% fewer tokens than AFlow at 1,000 queries on MBPP (4,802 vs. 10,519).

How the per-call numbers fit together:

  • Table 2 reports output tokens per call. Sec. 5.3 says so for the Scout, and it has to be so: the Scout reads the full query and the current output (App. E.1), so its 76.4 tokens cannot include input. AFlow therefore makes about 5.4 agent calls per query (6,643 / 1,238.5).
  • DynaMAS's calls follow Table 8: one first-agent call, a refine step on CONTINUE (40.4% of checkpoints), and one short corrective workflow on AUGMENT (24.1%), over about 1.2 checkpoints per query. The Judge runs at 59.6% of checkpoints (EARLYEXIT + AUGMENT), or about 0.72 calls per query, which gives 0.715 × 1,125.7 ≈ 805 output tokens.
  • DynaMAS's agent calls are shorter than AFlow's (≈807 vs. 1,238.5 output tokens per call), because its refine and fix steps edit an existing answer instead of sampling and ensembling new ones.

To be upfront: the Judge accounts for about a third of DynaMAS's per-query inference cost on MBPP (1,417 of 4,202 tokens), so we will soften the "modest overhead" wording in Sec. 5.3.

AuthorsOpenReview ↗

response to Reviewer jbLa (part 2)

W1. "The savings mostly reflect an over-provisioned baseline on saturated benchmarks."

Thank you for asking. We agree that AFlow is a bit over-provisioned on easy queries; that is the problem the paper sets out to solve. Our claim is that an arbitrary early exit cannot achieve low cost and high accuracy at the same time. Two results already in the paper show this:

  • Stopping after a single call (IO, Table 1) cuts AFlow's output tokens by 85–92%, but loses 14.9 / 3.3 / 4.2 points on MBPP / HumanEval / GSM8K and 9.9 F1 on HotpotQA.
  • Exiting on Scout signals alone (Scout-only, Table 4, HotpotQA) cuts AFlow's tokens by 25.7% (9,548 vs. 12,849) and latency by 81.7%, but F1 drops to 37.24. That is below AFlow (38.72) and 6.4 points below the full cascade (43.63), and it even uses 10% more tokens than the full cascade (9,548 vs. 8,648).

You are right that DynaMAS uses about 4× IO's output tokens on HumanEval and about 6× on GSM8K. Seen from the other side, AFlow uses about 2.7× DynaMAS's output tokens on MBPP (6,643 vs. 2,430) and is still less accurate. To check that DynaMAS does not only look good next to an over-provisioned pipeline, we also compared it with workflows that are not over-provisioned: the best-scoring 2-agent and 3-agent workflows found during AFlow's own search on each benchmark. On MBPP, these are generate → reflect and generate → verify → reflect.

Each cell shows accuracy (F1 for HotpotQA) · output tokens · latency in seconds.

MethodMBPPHumanEvalGSM8KHotpotQA
IO (Table 1)79.3 · 1,005 · 19.592.6 · 1,425 · 23.591.7 · 1,343 · 25.528.8 · 1,325 · 74.5
Best 2-agent workflow from AFlow's search85.6 · 2,810 · 54.693.2 · 3,990 · 65.893.0 · 3,760 · 71.436.0 · 3,710 · 208.6
Best 3-agent workflow from AFlow's search87.1 · 4,520 · 87.893.9 · 6,410 · 105.893.7 · 6,040 · 114.837.2 · 5,960 · 335.3
AFlow (Table 1)94.2 · 6,643 · 134.395.9 · 9,391 · 171.695.9 · 17,583 · 198.338.7 · 12,849 · 515.3
DynaMAS (Table 1)94.7 · 2,430 · 78.096.0 · 5,877 · 108.396.3 · 8,624 · 125.243.6 · 8,648 · 125.4

Small fixed workflows improve on IO, but fixing the workflow size in advance does not give a good trade-off:

  • MBPP: DynaMAS beats the 2-agent workflow by 9.1 points with fewer tokens (the 2-agent workflow is faster: 54.6 s vs. 78.0 s), and beats the 3-agent workflow by 7.6 points with about half the tokens and lower latency.
  • HotpotQA: DynaMAS is 6.4 F1 above the 3-agent workflow and 2.7× faster, although it uses 1.45× as many output tokens.
  • HumanEval / GSM8K: here the small workflows are competitive on cost. On HumanEval, the 2-agent workflow is cheaper than DynaMAS and the 3-agent workflow costs about the same (6,410 vs. 5,877 tokens; 105.8 vs. 108.3 s). On GSM8K, both are cheaper. However, all of them stay 2.1–3.3 points below DynaMAS, and below AFlow as well.

Across the four benchmarks, DynaMAS is the only system in this table besides AFlow itself that reaches AFlow-level accuracy, and it does so at well below AFlow's cost. Getting both requires deciding per query rather than fixing the workflow size up front.

Comparison with cheap baselines. DynaMAS is a routing layer, so its natural reference is the workflow it runs on. Still, it also holds up against the cheaper baselines in Table 1:

  • MBPP: fewer output tokens than every method except IO (2,430 vs. 3,274 for Flow, the next cheapest). Its total of 4,202 tokens is also below CoT's 4,340 (Table 7).
  • HotpotQA: +14.8 F1 over IO and +4.3 over the best baseline (MaAS, 39.3), with fewer output and total tokens than every baseline except the single-call IO and CoT.
  • HumanEval / GSM8K: IO is cheaper, as you point out. So we gave IO the same output-token budget as DynaMAS and spent it on extra samples: best-of-n for code, with samples selected using the same prompt-visible tests that every method sees, and self-consistency for math.
Same output-token budget as DynaMASMBPP (2,430)HumanEval (5,877)GSM8K (8,624)
IO, best-of-n (code) / self-consistency (math)84.5 ± 0.4 (n = 2)94.8 ± 0.3 (n = 4)93.8 ± 0.3 (n = 6)
DynaMAS94.796.096.3

Extra samples at equal cost do not close the gap. On HumanEval and GSM8K, the gap is 1.2–2.5 points: clear on GSM8K's 1,319 test problems, but only suggestive on HumanEval's 131. This is why our accuracy claim rests on the harder benchmarks in Q4. Sampled IO also matches or beats the fixed 3-agent workflow on these two sets (94.8 vs. 93.9 on HumanEval; 93.8 vs. 93.7 on GSM8K), which is another sign that a fixed small pipeline is not an efficient way to spend tokens.

AuthorsOpenReview ↗

response to Reviewer jbLa (part 3)

W2 / Q3. Is the accuracy gain due to the Judge and the Summarizer-written workflows?

The synthesized workflows are actually part of the method (contribution 3). And to eliminate the effect of the additional workflow pool and isolately eveluate the routing performance, we did additional experiments about other baselines using the same final workflow pool and the same Judge:

MethodHotpotQA F1 / total tokensMBPP Acc. / total tokens
AFlow38.7 / 18,27394.2 / 10,519
AFlow + the most-used pool workflow, always appended40.5 ± 0.6 / 22,70094.4 ± 0.3 / 15,400
AFlow + same Judge, pool, and tree prior, choosing a fix after the final step41.5 ± 0.6 / 21,60094.6 ± 0.3 / 14,000
MasRouter + same pool (A0 only, Table 10: 32.2 / 89.8)37.5 ± 0.7 / 17,50093.0 ± 0.5 / 8,500
DynaMAS43.6 / 12,77394.7 / 4,202

In the post-hoc variant (third row), the Scout runs once on AFlow's final output, so the Judge sees the same features and tree prior as in DynaMAS. DynaMAS still leads it by 2.1 F1 on HotpotQA (paired bootstrap, 95% CI +1.5 to +2.7 over the 7,405 test questions) while using about 40% fewer tokens. On MBPP, the two AFlow variants tie DynaMAS on accuracy (MasRouter + pool is 1.7 points lower), and DynaMAS is 2–4× cheaper than all three. Deciding mid-run matters for two reasons: the fix is applied where the failure first shows up (for example, a missing second hop is caught before the answer is written), and easy queries exit early. Table 10 shows the same with the seed pool only: DynaMAS-A0 beats MasRouter by 1.5 points on MBPP and 6.3 F1 on HotpotQA, with 38% and 10% fewer tokens.

On Table 4. Judge-only is the "Judge at every checkpoint" ceiling. The cascade reaches 99.3% of its F1 (43.63 vs. 43.94) with 71% lower latency (125.4 s vs. 437.5 s) and 31% fewer tokens. This is why the Scout exists.

On Appendix C.5 (86.3 → 92.1 on MBPP, 36.1 → 42.0 F1 on HotpotQA). These numbers were measured on the validation warm-up stream, starting from A0 with a cold tree. That is why their starting points are below even the A0-only test results in Table 10 (91.3 and 38.5). They track progress during warm-up, not the final gain on the test set.

Q4. Does accuracy hold on harder, non-saturated benchmarks?

Yes. We added all 1,324 level-5 test problems from MATH and the 2,417-question MuSiQue-Ans dev set. Validation data is drawn from each benchmark's training split, the executor is the same, and all state is frozen before testing.

BenchmarkIOCoTAFlowDynaMASΔ output tokens vs. AFlowΔ latency vs. AFlow
MATH (level 5), Acc.44.048.555.0 ± 0.658.0 ± 0.6−30%−30%
MuSiQue, F121.023.528.0 ± 0.632.0 ± 0.6−25%−60%

Both gains are statistically significant (95% CIs: +1.4 to +4.6 on MATH and +2.8 to +5.2 on MuSiQue). Token savings are smaller than on the four original benchmarks (25–30% here vs. 33–63% in Table 1) because fewer queries can exit early. With every role set to the executor (as in Q1), DynaMAS keeps +2.4 and +3.2 points of these gains, so the attribution result from Q1 also holds on the harder benchmarks.

We hope these results address your concerns. In the revision, we will clarify that the Judge is "stronger" only relative to the Scout, and we will soften the "modest overhead" claim in Sec. 5.3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.