- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DynaMAS controls multi-agent workflow execution using intermediate outputs. A lightweight Scout combines deterministic checks and a small LLM to label each checkpoint output CONTINUE, EARLYEXIT, or AUGMENT, a stronger Judge confirms proposed exits or selects corrective workflows, and a background Summarizer updates a failure-type mapping and synthesizes new workflows from completed traces. Across four benchmarks, the paper reports average reductions of 46.1% in output tokens and 47.8% in latency relative to AFlow, with preserved code and math accuracy and improved HotpotQA F1.
Critical strengths
-
Mid-execution routing is a well-motivated idea. The paper shows that a static workflow over-provisions easy queries and under-provisions hard ones (Section 3), and its insight that the need for more computation often appears only mid-workflow sets it apart from routers that decide once per query. Deciding at checkpoints from intermediate outputs, with a lightweight Scout that escalates to a stronger Judge, lets one system both exit early and add corrective workflows, and it layers onto an existing searched workflow, calibrating its thresholds on validation data instead of training a separate router.
-
The design is supported by controlled evidence. Ablations show that, on HotpotQA, the two-tier cascade nearly matches judging every checkpoint at a fraction of the latency, and that removing exit verification lowers task performance on every benchmark; the token and latency savings over AFlow are also large and far exceed run-to-run variation while code and math accuracy stay at AFlow's level. This makes the efficiency claim credible, subject to the issues below.
Critical weaknesses
-
The experiments do not yet show that the efficiency comes without loss. The headline claim is that DynaMAS cuts tokens and latency while keeping accuracy, and two pieces of evidence for it are missing. First, exit reliability is not measured at the deployed threshold. The Scout's best MBPP operating point has 65.7% precision (Section 5.4), while the Judge confirms 90.1% of MBPP exit proposals (Table 8); if both figures describe the same proposals, more than a quarter of confirmed exits would be incorrect, which would need to be reconciled with MBPP accuracy matching AFlow. Second, it is unclear whether the system learns from test labels. Appendix C freezes calibration on validation data before testing, but Section 4.2 says the failure mapping and candidate set adapt continuously as queries arrive, and Algorithm 2 learns from verified outcomes that Appendix E.3.1 defines using gold answers and failing unit tests. Measuring exit precision at the deployed threshold, and rerunning with all state frozen after validation, would settle both points.
-
The reported savings may not include the full controller cost. Table 2 says the Judge is invoked at 15.4% of checkpoints, but the Judge is called on every EARLYEXIT or AUGMENT decision (Section 4), and Table 8's 35.5% EARLYEXIT plus 24.1% AUGMENT on MBPP implies Judge calls at 59.6% of checkpoints. Table 2 likewise lists the Scout at 45.7%, although the Scout runs at every checkpoint. Since each Judge call costs about as many tokens as a workflow agent (Table 2), and the paper does not say which Scout, Judge, Summarizer, and warm-up costs Tables 1 and 7 include, the 46.1% output-token and 47.8% latency reductions cannot yet be read as full inference-cost savings.
Questions for authors
- Evaluation protocol: Which structures update during held-out testing, do those updates use the verified outcomes defined in Appendix E.3.1, and when do those updates occur relative to warm-up?
- Tree and accounting: What counter and initialization control splitting, how is the failure posterior normalized, and does the mapping, which gives every checkpoint the final-run label, predict current corrective needs or eventual run failure? How do Table 2's rates reconcile with Algorithm 1 and Table 8, which Scout, Judge, Summarizer, and warm-up costs do Tables 1 and 7 include, and how are failed selections counted in routing quality?
- Frozen state and exit reliability: Do the results hold with all state frozen after validation? At the deployed thresholds, what is exit precision, how often are confirmed exits incorrect, and how often does DynaMAS fail where the original workflow succeeds?
- Attribution: The Judge has adaptive access to seven code and eight RAG seed workflows plus synthesized ones, and the Judge-only variant already reaches 43.94 F1 on HotpotQA against the always-execute baseline's 38.72 F1 (Table 4). How does AFlow perform with the same workflow pool and matched controller access?
- Positioning: Section 3 states that prior methods commit to a single fixed workflow for all queries, yet MaAS and MasRouter, both compared in Table 1, already build query-dependent systems. The Scout-then-Judge escalation also follows LLM cascades such as AutoMix (Aggarwal et al., 2024, "AutoMix: Automatically Mixing Language Models") and the cascaded judges of Trust or Escalate (Jung et al., 2025, "Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement"), which Section 2.2's brief contrast with cascades does not address, and MASC (Shen et al., 2026, "Metacognitive Self-Correction for Multi-Agent System via Prototype-Guided Next-Execution Reconstruction") detects and corrects step-level errors during execution. How does DynaMAS differ from these works, and is its specific addition per-checkpoint three-way routing with an online-grown workflow library?
5 comments
response to Reviewer Rjti (part 2)
W2 / Q2. Table 2 rates and cost accounting.
Thank you for checking this so carefully, and we apologize for the misleading caption. The invocation rates in Table 2 are relative to all agent calls of the baseline AFlow workflow on the same queries. The Scout does run at every DynaMAS checkpoint, but DynaMAS reaches far fewer checkpoints than AFlow makes agent calls, because it exits early or replaces the remaining pipeline with one short corrective workflow. That is why the Scout's rate is 45.7% rather than 100%. We will correct the misleading caption.
On cost, the reductions in Tables 1 and 7 are full inference-cost savings. Both tables include every call on the inference path: workflow agents, Scout, and Judge (including the Judge's reasoning tokens). Table 2 gives each unit's tokens per call, and Table 7 (App. C.2) gives the complete input, output, and total accounting. Per MBPP query, DynaMAS's cost breaks down by unit as follows:
| MBPP, tokens per query | Input | Output | Total |
|---|---|---|---|
| Workflow agents | 817 | 1,533 | 2,350 |
| Scout | 343 | 92 | 435 |
| Judge | 612 | 805 | 1,417 |
| DynaMAS, inference (sum = Table 7) | 1,772 | 2,430 | 4,202 |
| DynaMAS, amortized over 1,000 queries | 4,802 (4,202 inference + 600 warm-up) | ||
| AFlow (Table 7) | 3,876 | 6,643 | 10,519 |
The summarizer and the pilot run tokens are for one-time only. And these token consumptions can be amortized over the future runs. Even adding this additional overhead, our framework still has substantial token consumption reduction compared to baselines.
- Summarizer and warm-up. All Summarizer calls happen during warm-up, so we report them, together with the pilot runs and the warm-up stream, as a one-time cost of about 600K tokens on MBPP. Over 1,000 queries, this adds only around 600 tokens per query (the amortized row above).
Q2. Tree details.
- Counter and splitting. Each node keeps decayed counts n_v ← (1 − η_M) n_v + e_τ, with N_v = Σ_τ n_v,τ. App. A mistakenly prints η_M on the one-hot term; as printed, N_v could never exceed 1, so the N_v ≥ 50 split rule could never fire. We will fix this typo. With η_M = 0.01, N_v saturates at 100 and passes 50 after about 69 observations. The node then splits if the best threshold's KL gain exceeds 0.1, and a child is pruned when its KL divergence from its parent stays below 0.02 over a sliding window. The tree is initialized from pilot labels plus a Dirichlet prior α.
- Normalization. Each node term (α_τ + n_v,τ) / (α_0 + N_v), with α_0 = Σ_τ α_τ, is a distribution over failure types. We set λ_0 = 1 at the root and λ_j = N_vj / (N_vj + κ) below it, so the weights ω_j sum to 1.
- What the tree predicts. Each checkpoint is labeled with the run's final failure type, so the tree predicts "what this run will eventually fail on, given what we see now." This is the prior the Judge needs. Updating only from the Summarizer's evidence_step onward (App. E.3.1) gives +1.5 Top-1 on HotpotQA (74.7 → 76.2) with the same F1.
- Failed selections count as misses for Top-1 and Top-2 in Table 3, and as Q = 0 when computing Q_t. With this convention, the mean Q_t on HotpotQA is 0.88.
response to Reviewer Rjti (part 3)
Q4. Attribution. We have done additional evaluation to show that even if the baselines were given the same additional workflow pool, our routing design can still have higher accuracy and lower token consumption. We gave the baselines the same final workflow pool, and gave AFlow the same Judge and tree prior as well:
| Method | HotpotQA F1 / total tokens | MBPP Acc. / total tokens |
|---|---|---|
| AFlow | 38.7 / 18,273 | 94.2 / 10,519 |
| AFlow + the most-used pool workflow, always appended | 40.5 ± 0.6 / 22,700 | 94.4 ± 0.3 / 15,400 |
| AFlow + same Judge, pool, and tree prior, choosing a fix after the final step | 41.5 ± 0.6 / 21,600 | 94.6 ± 0.3 / 14,000 |
| MasRouter + same pool (A0 only, Table 10: 32.2 / 89.8) | 37.5 ± 0.7 / 17,500 | 93.0 ± 0.5 / 8,500 |
| DynaMAS | 43.6 / 12,773 | 94.7 / 4,202 |
In the post-hoc variant (third row), the Scout runs once on AFlow's final output, so the Judge sees the same features and tree prior as in DynaMAS. DynaMAS still leads it by 2.1 F1 on HotpotQA (paired bootstrap, 95% CI +1.5 to +2.7) with about 40% fewer tokens. On MBPP, the AFlow variants tie DynaMAS on accuracy (MasRouter + pool is 1.7 points lower), and DynaMAS is 2–4× cheaper than all three. Deciding mid-run matters because the fix lands where the failure first shows up (for example, a missing second hop is caught before the answer is written), and easy queries exit early. To isolate this with identical components, we also restricted the same controller to a single decision before execution: it then reaches 40.0 F1 on HotpotQA, vs. 43.6 when it decides at every checkpoint.
On Table 4, Judge-only is the "Judge at every checkpoint" ceiling. The cascade reaches 99.3% of its F1 (43.63 vs. 43.94) with 71% lower latency (125.4 s vs. 437.5 s) and 31% fewer tokens.
Q5. Positioning.
- MaAS and MasRouter are query-dependent, but they decide before execution. We will correct Sec. 3 accordingly.
- AutoMix and Trust or Escalate escalate between models to produce one answer or one judgment. We adopt their cascade principle (and, like Trust or Escalate, conformal thresholds), but DynaMAS routes workflow actions inside a multi-agent trace. We will cite and credit both.
- MASC corrects anomalous steps with a single correction agent, with reliability as its goal. It has no cost-driven exit, and its correction action is fixed.
- Our specific addition is as you describe it: per-checkpoint exit / continue / augment routing, with a Judge-verified exit and an online-grown, failure-typed workflow library, training-free and layered on an existing workflow. As a direct check, an AutoMix-style self-verification exit reaches 93.0 at about 3,600 output tokens on MBPP and 38.0 F1 at about 10,500 on HotpotQA, vs. 94.7 at 2,430 and 43.6 at 8,648 for DynaMAS.
Thank you again. In the revision, we will correct the misleading words and add additional evaluations to the paper.
response to Reviewer Rjti (part 1)
Response to Reviewer Rjti
Thank you for the very detailed review. We will answer your questions and provide additional evaludation to support our claim.
W1a / Q3. Exit reliability, and 65.7% vs. 90.1%.
The two figures describe different sets of exit proposals. The 65.7% comes from the diagnostic sweep in Sec. 5.4, not from the deployed exit rule, and the two differ in three ways:
Measured at the deployed thresholds:
This confirms that early_exit doesn't affect the correctness. 96% of confirmed MBPP exits are correct, and an exit loses an answer that AFlow gets right on only 1.5% of queries, outweighed by 2.0% the other way. On HumanEval and GSM8K, confirmed-exit precision is 96.5% and 96%, with regressions vs. rescues of 1.2% vs. 1.3% and 1.0% vs. 1.4% (Table 1's +0.1 and +0.4). The HotpotQA net counts F1 ≥ 0.5 as correct; under exact match it is still +4.1. We will add the deployed operating point to Sec. 5.4 and Fig. 6.
The savings are not just truncation. On CONTINUE and AUGMENT, DynaMAS runs one refine step or one short corrective workflow instead of AFlow's full remaining pipeline. But with routing disabled (first agent, one refine, then stop), accuracy falls to 84.0 ± 0.4 on MBPP and 35.5 ± 0.6 F1 on HotpotQA, about Self-Refine's level (Table 1). Removing exit verification costs 2.9 points on average for only a 5.8% token saving (Table 9).
W1b / Q1 / Q3. Does the system learn from test labels?
No. Gold outcomes (App. E.3.1) are used only during the validation warm-up, which seeds the failure-type tree and A_syn; this is the phase shown in Fig. 8. In the reported runs, the tree, pool, and thresholds are then frozen, and Algorithm 2 (including its reliability decay) does not run at test time. The only state that changes at test time is the φ_hist reliability EMA, which is driven by Judge confirmations and the deterministic checkers, never by gold labels. We will fix the "adapts continuously" wording in Sec. 4.2. We also reran all four benchmarks under three stricter settings:
The label-free warm-up replaces gold outcomes with executor-written tests (code), self-consistency (math), and the Judge's evidence-entailment check (RAG). It never uses pass/fail results on the benchmark's own asserts. At inference time, all methods, AFlow included, see the same prompt-visible tests (AFlow's protocol; App. G). The fully frozen run is within one standard deviation of Table 6, and every change is at most 0.8 points (at most 1.5 standard deviations).
Data splits. MBPP and GSM8K use their official test sets (500 and 1,319 problems). HotpotQA's test labels are hidden, so "test" is the 7,405-question full-wiki dev set. HumanEval has no train or validation split, so we hold out 33 problems for calibration and test on the remaining 131. We will state all of this in Sec. 5.1.