acceptodds
Under review as a conference paper at ICLR 2027

Pay for the Fault, Not the Flow: Label-Free In-Flow Multi-Agent Workflow Optimization

Abstract

Large language models (LLMs) increasingly construct multi-agent workflows that decompose a complex task and assign specialist agents from a pool. However, building such a workflow well remains challenging: how finely to divide the task, which agent to trust with each subtask, and when to create a new specialist are all critical decisions a workflow constructor needs to settle up front. Thus, whether each subtask succeeds remains unknown until the workflow runs. Yet, improving a workflow is costly. Locating a fault usually requires a reference answer, a graded outcome, or a trained assessor, and the fix is applied to the whole workflow through re-execution, re-search, or retraining. We propose InFlowOp, which prices every decision in one label-free cost that weighs how well an agent's competence meets what a subtask demands against how much that agent takes to run. Before execution, InFlowOp bidirectionally determines the granularity of task decomposition and agent assignment following from the cost rather than from a fixed template. During execution, InFlowOp corrects a fault with the cheapest move via the same cost that serves the workflow both as it is built and as it runs. Facing the workflow-level evaluation challenge, we introduce Braid, a benchmark whose tasks require multi-agent coordination beyond single-agent capability. Across various domains and backbones, InFlowOp outperforms single agent baselines by up to , achieving with in-flow optimization.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
workflow optimization, multi-agent workflow, multi-agent system, multi-agent coordination

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Gui4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

INFLOWOP combines cost-guided workflow construction with local repair during execution. A shared estimator scores agent suitability and checks subtask conditions. The paper also introduces BRAID, which combines existing questions with distractor sources, and reports improvements across several backbones.

Critical strengths

Jointly considering task granularity and agent suitability is a useful direction. The consistent improvements over single-agent baselines suggest practical value in the resulting workflow construction.

Critical weaknesses

The methodological advance over existing workflow systems appears limited. AFlow (Zhang et al., 2025) already automates workflow search, while DyFlow (Wang et al., 2025) adapts workflows using intermediate feedback. The proposed cost-guided construction and local repair offer a limited extension, whose central rubric estimator is insufficiently specified. It remains unclear how scores are computed from task requirements, agent descriptions, and execution records, or why they should represent success probabilities without calibration. Using the same estimator to accept repairs also does not establish that those repairs improve final correctness. References

The empirical gains do not convincingly justify the overhead or establish an advantage over existing workflow methods, which are not directly compared. In-flow repair adds only 1.59 percentage points for Qwen3.5-9B, while Figure 9 shows roughly 20–30 times the single-agent token consumption for GPT-5-mini and GPT-5.4-mini. Matching turns does not resolve this resource imbalance. Furthermore, BRAID bundles questions and distractor sources without introducing dependencies between the questions. This tests source retrieval and multi-question processing, but does not substantiate the claimed need for interdependent multi-agent reasoning beyond single-agent capability.

Zhang et al. (2025). AFlow: Automating Agentic Workflow Generation. Wang et al. (2025). DyFlow: Dynamic Workflow Framework for Agentic Reasoning.

Questions for authors

How exactly is the rubric estimator implemented, and what supports interpreting its scores as success probabilities and using them to validate repairs without additional training?

What substantive capability does INFLOWOP add beyond existing workflow search and feedback-driven replanning, particularly given its token overhead?

What in BRAID requires interdependent reasoning rather than locating the relevant sources and answering each question separately?

4 comments
AuthorsOpenReview ↗

Author Response (1/2): Thank you for your valuable review and insightful feedback

Thank you for your careful review and constructive feedback, and for recognizing that our work is useful and our consistent improvements suggest practical value. We hope to further address your remaining concerns below.

W1+Q2: Advance over existing workflow methods

Thank you for your insightful comments.

1. Direct comparison (W2 "which are not directly compared", W1 "AFlow (Zhang et al., 2025) already automates workflow search"). We add AFlow [1] and DyFlow [2] from their official code. As released, both are text-in/text-out and cannot open BRAID's documents and images, so we run both with Qwen3.5-9B and the same tools, skills, sandbox, budget, and run limits as INFLOWOP, on five datasets of Tab. 1. Mean accuracy (%) per dataset and tokens per task below show that COALESCE alone is already higher than both methods on every dataset with fewer tokens, and INFLOWOP is the highest:

MethodMMLongBench-DocDocFinQAChartMuseumOlympiadBench-Math (Text)OlympiadBench-Physics (Text)Tokens / task
AFlow [1]10.3316.1026.9133.144.20382.7k
DyFlow [2]8.5018.1131.3437.305.63346.9k
greedy-search8.8513.4617.6337.674.53263.8k
COALESCE12.1921.2538.5546.478.70289.4k
INFLOWOP14.8125.2041.2849.9710.88334.8k

2. What INFLOWOP adds (Q2 "What substantive capability does INFLOWOP add beyond existing workflow search and feedback-driven replanning"). AFlow improves a workflow offline over graded tasks, and DyFlow re-plans its next steps from intermediate feedback. INFLOWOP adds two workflow-level capabilities beyond these. Before execution, one label-free cost decides the granularity of the workflow, the agent of each subtask, where a new specialist is needed, and creating the specialist that a subtask demands (§3.1). During execution, a fault is repaired at the subtask where it arises, every verified result is kept, and no reference answer, re-run of the workflow, or training is needed (§3.2). Results above show this substantive capability gap empirically: COALESCE alone already exceeds both methods on every dataset, and INFLOWOP exceeds the stronger of the two by up to +12.67. We will add this comparison and discussion to our revision.

W1+Q1: Estimator specification and repair validation

Thank you for your detailed questions.

1. Implementation (W1 "how scores are computed from task requirements, agent descriptions, and execution records", Q1 "How exactly is the rubric estimator implemented"). Our rubric estimator implements match without any LLM call (§3.1). A frozen sentence encoder (all-MiniLM-L6-v2) embeds a demanded text and a supplied text. An increasing function then maps their cosine similarity to p^∈[0,1]\hat p \in [0, 1]. In construction, the demanded text is what a subtask requires, and the supplied text is the agent's card (Agent Card box, p.19). Where a subtask declares several conditions, the weakest one sets the score. In the full INFLOWOP, each card is combined with the agent's self-evolving profile (Fig. 5), a kernel-weighted average over its past execution records, whose weight grows as records accumulate. In detection, the estimator scores each declared condition against the realized input or output (Eq. 3).

2. Scores without calibration (W1 "why they should represent success probabilities without calibration", Q1 "what supports interpreting its scores as success probabilities"). Every decision INFLOWOP makes from the scores depends only on how they compare with one another or with a fixed bar: each subtask is bound to its lowest-cost agent (Alg. 3), agent creation is triggered when no agent clears the competence bar Cˉ\bar C (Eq. 1), a condition is unmet once it reaches the unmet bar uˉ\bar u (Eq. 3), and re-assignment takes the next-best agent (Eq. 20). Rescaling the scores and bars in any way that preserves their order therefore leaves every decision unchanged, so none requires calibrated probabilities. We further validate the scores on a small random sample from the Qwen3.5-9B run, with human labeling. At the configured bars, the reliability estimates predict subtask success with 90.48% precision and 86.36% recall over 50 subtasks, and violation detection finds unmet conditions with 87.80% precision and 85.71% recall over 50 conditions.

AuthorsOpenReview ↗

Author Response (2/2): Thank you for your valuable review and insightful feedback

3. Repairs and final correctness (W1 "does not establish that those repairs improve final correctness", Q1 "using them to validate repairs without additional training"). A repair is kept only if it scores strictly higher than the original on the same detector scale, so a tie or an unscoreable repair keeps the original. On final answers over the 16 arms of Tab. 1 with GPT-5.6-luna, a repair is accepted on 826 tasks, and on them the final score improves on 85.23% (Δ ≥ 0, Δ: the change from COALESCE alone) and worsens on 14.77% (Δ < 0). Accepted repairs thus improve the final score on the large majority of their tasks, without any training.

W2+Q2: Gains and token overhead

Thank you for your thoughtful comments.

1. The in-flow gain (W2 "In-flow repair adds only 1.59 percentage points for Qwen3.5-9B"). Pairing each task across COALESCE and COALESCE with in-flow optimization on Qwen3.5-9B, the +1.59 has a bootstrap 95% confidence interval of [+1.24, +1.94] and is significant on every benchmark (p < 0.01). It improves 93.72% of tasks (Δ ≥ 0) and worsens 6.28% (Δ < 0), at matched turns and without reference answers or re-runs. Best-of-N instead repeats the whole workflow four times and keeps the outcome with the highest graded score, yet it stays below INFLOWOP on every backbone (§E, Fig. 9).

2. Token overhead (W2 "roughly 20–30 times the single-agent token consumption", W2 "Matching turns does not resolve this resource imbalance", Q2 "particularly given its token overhead"). Across the five backbones of Fig. 9, INFLOWOP's tokens per task span 1.8× (339k to 608k), while the Single Agent's span 27× (12.7k to 347k). The ratio thus follows how the Single Agent's spending varies across backbones. We further give the Single Agent of Tab. 1 INFLOWOP's token budget on each task instead of its turn budget, on Qwen3.5-9B and on GPT-5.4-mini. Accuracy (%) over the 16 arms below shows that INFLOWOP stays far higher at the same token budget:

BackboneSingle Agent, turn budgetSingle Agent, INFLOWOP's token budgetINFLOWOP
Qwen3.5-9B17.3815.2425.13
GPT-5.4-mini17.4915.7229.46

On GPT-5.4-mini, the Single Agent spends only 39.0k tokens per task against a budget of 339.0k, so a larger budget does not close the gap. With the same tools, AFlow and DyFlow also spend more tokens per task than INFLOWOP (W1+Q2). On Qwen3.5-9B, correction, including profile updates, takes 17.7% of the tokens and buys the +2.92 of INFLOWOP over COALESCE alone (Tab. 2).

W2+Q3: Interdependence in BRAID

Thank you for your insightful question.

1. What BRAID requires (W2 "without introducing dependencies between the questions", Q3 "What in BRAID requires interdependent reasoning"). Each BRAID task provides more sources than its questions need, without stating which source serves which question (§C.2). In a Distributed task, each of the kk questions has its own gold source among 2k−12k-1 sources. So sources have to be assigned to questions jointly, since the source that answers one question answers no other. In an Anchored task, all kk questions are answered from one gold source among kk sources, so every part depends on first locating that source. Fig. 7 (§C.1) shows the effect: on the original datasets, a single agent is more accurate than a workflow an LLM constructs (54.37% vs. 42.35%), while after BRAID composition the single agent drops to 10.82% and the workflow reaches 48.99%.

2. Answering each question separately (Q3 "rather than locating the relevant sources and answering each question separately"). We add a single agent that answers each question of a task in its own run, with the same tools, skills, and sandbox, under matched token budgets. With Qwen3.5-9B on the 16 arms (Tab. 1), it reaches 19.39%, stronger than the Single Agent's 17.38% in Tab. 2, yet INFLOWOP's 25.13% is +5.74 higher, and higher in each of the 16 arms. Such a split still leaves each question to locate its own gold source among all the sources of its task, which is the routing that BRAID composition introduces.

References

[1] J. Zhang et al. AFlow: Automating Agentic Workflow Generation. 2025.

[2] Y. Wang et al. DyFlow: Dynamic Workflow Framework for Agentic Reasoning. 2025.

Thank you for your valuable review and constructive feedback. We sincerely hope our response and clarifications above can meaningfully address your remaining concerns, and we will make sure to apply them to our revisions to further enhance our paper quality.

Thank you for the thoughtful response and the effort put into the additional experiments. I appreciate these improvements, though my main concerns remain. I will keep my score of 2.

AuthorsOpenReview ↗

Author Response: Thank you for your valuable review and follow-up feedback

Thank you very much for your thoughtful follow-up and for taking the time to review our additional experiments and clarifications. We sincerely appreciate your recognition of our efforts and continued engagement with our work. We have sought to address the concerns raised in your review through additional experiments, analyses, and clarifications. If there are any specific points or concerns that you feel remain insufficiently addressed, we would be very grateful for the opportunity to provide further clarification. We sincerely appreciate your time and consideration.

Reviewer_Lc3bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes INFLOWOP, a training-free framework for constructing and optimizing multi-agent workflows. Its construction stage, COALESCE, selects decomposition granularity and agent assignments using a cost objective combining estimated reliability, latency, and agent creation. During execution, the framework detects violations of subtask input/output contracts and applies local corrections while reusing prior results. The paper also introduces BRAID for evaluating workflow-level coordination. Experiments report aggregate improvements over tool-enabled single-agent baselines across six backbones.

Critical strengths
  • The paper addresses a practical problem: correcting workflow failures while retaining useful intermediate computation.

  • Cost-driven atomic decomposition and coalescing provide an interesting way to jointly determine task granularity and agent assignments. Reusing the matching mechanism for runtime contract checks also gives the framework conceptual coherence.

  • The aggregate gains over the tool-enabled Single Agent baseline across six backbones are encouraging. Table 2 helps distinguish the contributions of construction, tempering, and evolving profiles.

Critical weaknesses

1. The core mechanism lacks reproducible detail, and its fault handling needs stronger justification.

The rubric and operational implementation of match are unspecified, while atomic decomposition, contract generation, and agent creation lack reproducible procedural details. Figure 3 compares end-to-end accuracy but does not directly validate reliability estimates or violation detection. Although Table 4 provides the competence and unmet-condition thresholds, their calibration and sensitivity are not examined.

Equation (4) assigns unmet inputs to decomposition, although these may also result from upstream failures. Algorithm 4's RE-DECOMPOSE branch splices in a new subgraph and returns without the execution check or rollback required by Algorithm 1. Moreover, Figure 8 presents a repaired output that violates its contract yet produces the correct final answer; this output would apparently be rejected by Algorithm 1. These inconsistencies leave both the acceptance procedure and the scope of the "never worse" guarantee unclear.

2. The profile protocol is unclear with respect to the label-free claim.

Figure 5 reports gains from retaining only "successful" experiences across runs, but the success criterion is not defined. If reference answers determine which experiences are retained, the label-free claim requires qualification. The paper should also specify which profile scope, evaluation order, and reset policy produce Tables 1 and 2, including whether information carries across evaluation tasks.

3. The evaluation lacks an established external workflow baseline and stronger validation of BRAID's coordination claims.

Workflow comparisons involve greedy construction, variants of the proposed framework, and best-of-N, whose implementation and outcome-selection procedure need clarification. The +9.64-point gain over greedy construction is reported only for Qwen3.5-9B and reflects the full framework rather than tempering alone.

The Single LLM baseline cannot access input files, as acknowledged in footnote 1, so the tool-enabled Single Agent is the more informative comparison. A stronger tool-enabled single-agent baseline under comparable token budgets would also help establish whether BRAID measures benefits from multi-agent coordination beyond increased workload and source-routing difficulty.

4. Matched turns do not establish matched compute, and uncertainty in the reported gains is not quantified.

Appendix E defines empirical problem-solving cost as total input and output tokens, but the experiments match turns. Figure 9 shows INFLOWOP using approximately 30× the Single Agent's tokens on GPT-5-mini and GPT-5.4-mini under the same 1× turn label; best-of-N, labeled 4× turns, uses approximately 100× the tokens on those models.

Although Appendix E states that total token accounting includes construction, execution, and correction, their separate costs and actual latency are not reported. Absolute token usage and latency should accompany accuracy. Confidence intervals or repeated-run results are also needed, particularly for the smaller incremental gains in Table 2: 1.59 percentage points from tempering and 1.33 from profiles.

5. COALESCE's pruning and finite-budget guarantees require clarification.

Appendix D.1 explains pruning through an optimistic lower bound, but does not specify how OPTIMISTICREMAINDER is computed or establish its admissibility when merging changes previously scored subtask costs. The paper should distinguish preservation of the optimum by pruning from any guarantee available when search terminates at the finite expansion budget BB. Construction time and solution quality as BB varies would help assess the practical trade-off.

Presentation Issues

  • Key definitions and algorithms needed to assess the method appear only in the appendix: the detailed cost terms in Eqs. (16)–(18), the ladder and closure in Eqs. (19)–(21), and Algorithms 3–4. Section 4 describes BRAID in a single paragraph, making the benchmark difficult to assess from the main paper.
  • Metaphorical terminology and section titles, such as "tempering" and "what earns the flow," sometimes obscure the technical content. More descriptive terminology would improve readability.
  • Accuracy differences are repeatedly written with "%" where percentage points are intended, including the abstract's +11.97% and +9.64%. This concern applies to reported differences, rather than accuracy values themselves.
  • Algorithm 4 reuses bb for both the blame label and a loop counter.
  • Typos and grammatical errors include "decomposes … and find" (§1), "multi-agent coordinations" (§1), "close-source" (§5.1), and "Tempter is confined to the the local subtask" (§D.1).
  • An unresolved editorial placeholder remains in Appendix C.

Reporting Inconsistencies

  • Table 1 and Figure 9 refer to 16 arms, whereas Table 3 lists 13 across the five reported domains: four Document, two Finance, two Chart, three Math, and two Physics arms. Please identify the included arms and provide the averaging formula for "All," since this affects interpretation and reproducibility of the aggregate results.
  • Section 5.2 states that all methods remain at or below 17.34% on Physics, but Table 1 reports 18.84% for GPT-5.6-luna under Single LLM.
  • The Table 1 caption should refer to the available complexity levels, since DocFinQA at k=5k=5 is excluded (§C.2).

Overall Assessment

The problem is relevant, the proposed construction and local-correction framework is interesting, and the aggregate results are promising. My main reservations concern reproducibility of the estimator and fault-handling mechanism, the unclear profile protocol, the lack of established workflow comparisons, and the mismatch between turn matching and token costs. Clarifying the algorithmic guarantees and evaluation protocol, together with a targeted baseline comparison under comparable resources, could materially improve my assessment.

Questions for authors
  1. Could you provide the rubric, scoring implementation, and procedures for atomic decomposition, contract generation, and agent creation? Could you validate reliability estimates and violation detection on a small annotated sample, including sensitivity to the two thresholds?

  2. How is a re-decomposed subgraph executed and validated before acceptance? How can the contract-violating repaired output in Figure 8 be accepted under Algorithm 1? Does "never worse" concern contract satisfaction, estimated cost, or final-answer accuracy?

  3. How is profile success determined? Are reference answers used for profile updates or workflow decisions beyond final evaluation? What profile scope, task order, and reset policy are used for Tables 1 and 2?

  4. Could you compare with one established workflow method and a stronger tool-enabled single-agent baseline under comparable tools and token budgets? How is best-of-N implemented, and how is its best outcome selected?

  5. What resource is matched per task? Please report absolute token usage and actual latency, separated into construction, execution, and correction, together with uncertainty estimates for the main and incremental gains.

  6. How is the pruning lower bound computed, what guarantee remains under the expansion limit BB, and how do construction time and solution quality change with BB?

5 comments
AuthorsOpenReview ↗

Author Response (1/3): Thank you for your valuable review and insightful feedback

Thank you for your careful review and constructive feedback, and for recognizing that our work addresses a practical problem and our gains hold across settings. We hope to further address your remaining concerns below.

W1+Q1+Q2: Mechanism specification and fault handling

1. Rubric and scoring (W1 "operational implementation of match", Q1 "the rubric, scoring implementation"). Our rubric estimator implements match without any LLM call (§3.1). A frozen sentence encoder (all-MiniLM-L6-v2) embeds a demanded text and a supplied text. An increasing function then maps their cosine similarity to p^∈[0,1]\hat p \in [0, 1]. In construction, the demanded text is what a subtask requires, and the supplied text is the agent's card (Agent Card box, p.19). Where a subtask declares several conditions, the weakest one sets the score. In the full INFLOWOP, each card is combined with the agent's self-evolving profile (Fig. 5), a kernel-weighted average over its past records whose weight grows as they accumulate. In detection, the estimator scores each declared condition against the realized input or output (Eq. 3).

2. Procedures (Q1 "procedures for atomic decomposition, contract generation, and agent creation"). Atomic decomposition splits the task into its finest atoms. Each atom states its input and output conditions, and the atoms form a data-flow DAG. COALESCE keeps only the atoms whose outputs reach the final answer through this DAG, and removes the others (Alg. 3). A subtask's contract consists of the input and output conditions of its member atoms (Alg. 3). Agent creation is triggered when no agent in the pool clears the competence bar Cˉ\bar C for a subtask (Eq. 1), charged with creation penalty γ\gamma (Alg. 3).

3. Validation and thresholds (W1 "does not directly validate reliability estimates or violation detection", Q1 "sensitivity to the two thresholds"). Following your suggestion, we validate both the reliability estimates and violation detection on a small random sample from the Qwen3.5-9B run, with human labeling. At the configured bars, the reliability estimates predict subtask success with 90.48% precision and 86.36% recall over 50 subtasks, and violation detection finds unmet conditions with 87.80% precision and 85.71% recall over 50 conditions. Varying each threshold with the other fixed (Qwen3.5-9B, 16 arms of Tab. 1), accuracy (%) below shows INFLOWOP is robust to both:

ThresholdLowerTab. 4Higher
Competence bar Cˉ\bar C24.88 (0.5)25.13 (0.7)25.74 (0.9)
Unmet bar uˉ\bar u24.78 (0.6)25.13 (0.8)25.12 (0.9)

4. Upstream causes and re-decomposition (W1 "upstream failures", Q2 "How is a re-decomposed subgraph executed and validated before acceptance?"). Every subtask is checked against its own contract as it completes, in topological order (Alg. 1). So an upstream subtask that underdelivers is tempered before its successors run. A flagged subtask is first re-assigned, re-running only that subtask, and is re-decomposed only when its original realization and every re-assignment produce no usable output. The re-cut subtasks then run in the same walk, each checked and tempered like any other, so a re-decomposition is validated by its execution and replaces only an unusable realization.

5. Fig. 8 (Q2 "How can the contract-violating repaired output in Fig. 8 be accepted under Algorithm 1?"). It is not accepted. As the Fig. 8 (§D.5) title states "The Worth of A Step Is Settled Downstream": scored against its own condition alone, a step that helps the flow reach the correct answer can be recorded as failed. It is a diagnostic on scoring a step in isolation, not a trace of a committed repair.

6. Acceptance and the guarantee (W1 "the scope of the 'never worse' guarantee", Q2 "Does 'never worse' concern contract satisfaction, estimated cost, or final-answer accuracy?"). A repair is kept only if it scores strictly higher than the original on the same detector scale, so a tie or an unscoreable repair keeps the original. "Never worse" therefore concerns contract satisfaction as the label-free detector scores it, with the progress already achieved retained (§3.2), not estimated cost or final-answer accuracy. On final answers over the 16 arms of Tab. 1 with GPT-5.6-luna, a repair is accepted on 826 tasks, and on them the final score improves on 85.23% (Δ ≥ 0, Δ: the change from COALESCE alone) and worsens on 14.77% (Δ < 0). Accepted repairs thus improve the final score on the large majority of their tasks.

W2+Q3: Profile protocol and the label-free claim

1. Profile success (W2 "the success criterion is not defined", Q3 "How is profile success determined?"). A profile record is written where a repair is attempted: the detector compares the original realization with the repaired one, and the agent whose realization scores higher is recorded as a success, the other as a failure. Fig. 5's "successful ones" are these records.

AuthorsOpenReview ↗

Author Response (2/3): Thank you for your valuable review and insightful feedback

2. Reference answers (Q3 "Are reference answers used for profile updates or workflow decisions beyond final evaluation?"). No. Reference answers only grade the final answer when we compute accuracy, and never enter a profile update, a cost, a detection, or an acceptance. The label-free claim therefore holds as stated.

3. Protocol for Tabs. 1-2 (W2 "whether information carries across evaluation tasks", Q3 "What profile scope, task order, and reset policy are used for Tabs. 1-2?"). Each run covers one arm and starts with an empty profile, so nothing carries across runs. Within a run, tasks follow their dataset order, and the profile is updated after every mm tasks (default m=20m=20): each batch of mm tasks is scored on the profile from earlier batches, held fixed within the batch, so no task informs its own score. Label-free records thus carry from earlier to later tasks of the same run, the within-run setting of Fig. 5, and across runs only in its across-run arms.

W3+Q4: Workflow and single-agent comparisons

1. Established workflow methods (W3 "lacks an established external workflow baseline", Q4 "compare with one established workflow method"). We add AFlow [1] and DyFlow [2]: Both are text-in/text-out and cannot open BRAID's documents and images, so we run both with Qwen3.5-9B and the same tools, skills, sandbox, budget, and run limits as INFLOWOP, on five datasets of Tab. 1. Mean accuracy (%) per dataset and tokens per task below show that INFLOWOP is the highest on every dataset, by +4.48 to +12.67 over the stronger of the two methods, while spending fewer tokens per task than either:

MethodMMLongBench-DocDocFinQAChartMuseumOlympiadBench-Math (Text)OlympiadBench-Physics (Text)Tokens/task
AFlow10.3316.1026.9133.144.20382.7k
DyFlow8.5018.1131.3437.305.63346.9k
greedy8.8513.4617.6337.674.53263.8k
INFLOWOP14.8125.2041.2849.9710.88334.8k

2. A stronger single agent under comparable token budgets (Q4 "a stronger tool-enabled single-agent baseline under comparable tools and token budgets"). We give the Single Agent of Tab. 1 the same tools, skills, sandbox, and INFLOWOP's token budget on each task instead of its turn budget. With Qwen3.5-9B on the 16 arms (Tab. 1), it reaches 15.24%, against 17.38% under the turn budget (Tab. 2) and 25.13% for INFLOWOP at the same token budget. Thus, more tokens do not close the gap, which comes from how INFLOWOP divides, assigns, and coordinates the parts (W3 "beyond increased workload and source-routing difficulty"): sources need to be assigned to questions jointly in a Distributed task, and every question depends on first locating one shared gold source in an Anchored task (§C.2). Fig. 7 (§C.1) shows the effect: after BRAID composition, a single agent drops from 54.37% to 10.82%, while a constructed workflow rises from 42.35% to 48.99%.

3. Best-of-N (Q4 "How is best-of-N implemented, and how is its best outcome selected?"). Best-of-N repeats workflow construction and execution N = 4 times per task, and keeps the outcome with the highest graded score. Its selection thus consults the reference answer, making it an upper bound for repeated construction. With four times INFLOWOP's turns, it still stays below INFLOWOP on every backbone (§E, Fig. 9).

4. The +9.64 gain (W3 "reflects the full framework rather than tempering alone"). We will state this more clearly in our revision: +9.64 is the full method's gain over greedy construction on Qwen3.5-9B (Tab. 2), which §5.2 decomposes into +6.72 from construction, +1.59 from in-flow tempering, and +1.33 from evolving profiles. Over the tool-enabled single agent, INFLOWOP gains +7.15 to +11.97 on all five backbones of Fig. 9 (§E).

W4+Q5: Matched resource, absolute cost, and uncertainty

1. Matched resource (W4 "Matched turns do not establish matched compute", Q5 "What resource is matched per task?"). Agent turns: on each task, the single agent receives the same turn budget as INFLOWOP (§E). Under matched tokens, INFLOWOP stays higher as well (W3+Q4).

2. Absolute tokens and latency by stage (W4 "Absolute token usage and latency should accompany accuracy", Q5 "separated into construction, execution, and correction"). We show the cost per task by stage over the 16 arms of Tab. 1, for Qwen3.5-9B and GPT-5.4-mini:

StageQwen3.5-9B tokensQwen3.5-9B LLM time (s)GPT-5.4-mini tokensGPT-5.4-mini LLM time (s)
Construction5.4k154.32.4k22.8
Correction64.8k616.251.4k163.5
Execution295.6k1,032.7285.3k326.4
Total365.8k1,803.2339.0k512.7
End-to-end latency (s)–1,963.8–702.6

Construction takes 1.5% and 0.7% of the tokens, and correction, including profile updates, 17.7% and 15.1%, showing INFLOWOP's +2.92 over COALESCE alone (Tab. 2).

AuthorsOpenReview ↗

Author Response (3/3): Thank you for your valuable review and insightful feedback

3. The 30× ratio (W4 "approximately 30× the Single Agent's tokens"). Across the five backbones of Fig. 9, INFLOWOP's tokens per task span 1.8× (339k to 608k), while the Single Agent's span 27× (12.7k to 347k). The ratio thus follows how the Single Agent's spending varies across backbones. Best-of-N's 100× has the same denominator, since it reruns the workflow four times (§E).

4. Uncertainty (W4 "Confidence intervals or repeated-run results are also needed", Q5 "uncertainty estimates for the main and incremental gains"). Pairing each task across the settings of Tab. 2 (Qwen3.5-9B), we show bootstrap 95% confidence intervals and the shares of tasks improved and worsened (Δ: a task's change in score) below:

Gain (Tab. 2)95% CIImproved (Δ ≥ 0)Worsened (Δ < 0)
+1.59, in-flow tempering[+1.24, +1.94]93.72%6.28%
+1.33, evolving profiles[+1.05, +1.61]94.23%5.77%
+9.64, full method over greedy construction[+8.77, +10.51]97.31%2.69%

Every interval excludes zero, and each gain is significant on every benchmark (p < 0.01).

W5+Q6: Pruning bound and the expansion budget

1. The lower bound (W5 "how OPTIMISTICREMAINDER is computed", Q6 "How is the pruning lower bound computed"). COALESCE prunes a partial coalescing once its lower bound reaches the cost of the best complete coalescing found (§D.1, Alg. 3). The bound is the cost committed so far plus the least cost the unplaced atoms can still incur: each unplaced atom contributes its cheapest single-atom cost, and each distinct new agent they require contributes one creation penalty. This bound is admissible whenever a merge does not lower the cost of the subtask it forms.

2. The expansion budget (W5 "distinguish preservation of the optimum by pruning", Q6 "what guarantee remains under the expansion limit B"). Pruning discards only partial coalescings that cannot complete below the incumbent, so under the condition above it keeps the least-cost coalescing. The expansion budget B = 20,000 (Tab. 4) separately bounds the expansions: if the frontier empties within B, COALESCE returns the least-cost coalescing, and otherwise its incumbent, which starts as the finest cut, one subtask per atom, and is replaced only by cheaper complete coalescings (Alg. 3). In our revision, we will state both guarantees clearly in §3.1 and §D.1.

3. Construction time and solution quality as B varies (Q6 "how do construction time and solution quality change with B?"). We vary B on Qwen3.5-9B over the 16 arms of Tab. 1, and show construction time per task and accuracy (%) below:

BConstruction time / task (s)Accuracy (%)
5,00025824.89
20,000 (Tab. 4)28625.13
100,00032525.76

A 20× larger budget adds 67 s per task and raises accuracy by +0.87, so INFLOWOP is robust to B, and a larger budget improves it further.

Presentation and consistency

Thank you for your thoughtful suggestions.

1. Arms and the average ("identify the included arms and provide the averaging formula"). Tab. 1 averages 16 arms, each a dataset at one composition and complexity level: MMLongBench-Doc (both compositions, kk = 3, 5) for Doc, DocFinQA (both compositions, kk = 3) for Fin, ChartMuseum (Distributed, kk = 3, 5) for Chart, and OlympiadBench-Math and -Physics, text and multimodal (Distributed, kk = 3, 5) for Math and Phys. Each domain column averages its arms, and All averages the 16 arms without weighting (13 for GPT-5.4, whose Doc covers the Distributed kk = 3 arm). We will show these clearly in our revision.

2. Physics ("all methods remain at or below 17.34% on Physics"). We will correct this sentence to "at or below 18.84%", the Single LLM on GPT-5.6-luna.

3. Presentation. Tempering is defined in the §3.2 box, and we will use more descriptive section titles. We will also write accuracy differences in percentage points, including the abstract's +11.97 and +9.64, rename the loop counter of Alg. 4, and fix the listed typos and the placeholder in App. C. As space allows, we will move Eqs. 16–21, Algs. 3–4, and a fuller description of BRAID into the main paper.

References

[1] J. Zhang et al. AFlow: Automating Agentic Workflow Generation. 2025.

[2] Y. Wang et al. DyFlow: Dynamic Workflow Framework for Agentic Reasoning. 2025.

Thank you for your valuable review and constructive feedback. We sincerely hope our response and clarifications above can meaningfully address your remaining concerns, and we will make sure to apply them to our revisions to further enhance our paper quality.

Reviewer_QPgDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes INFLOWOP for constructing and locally repairing multi-agent workflows without reference answers. It combines cost-guided task coalescing and agent assignment with execution-time contract checking and scoped re-execution. Experiments on the proposed BRAID benchmark report improvements over single-agent and workflow baselines.

Critical strengths
  1. The framework targets a practical source of overhead. Scoped repair and reuse of unaffected results can avoid unnecessary workflow re-execution.

  2. The evaluation covers multiple domains and backbones. Component ablations help distinguish gains from workflow construction, in-flow repair, and evolving agent profiles.

Critical weaknesses
  1. Contract satisfaction is not validated as a reliable correctness signal. Figure 8 explicitly shows that contract judgments can disagree with downstream correctness, undermining the claimed non-degradation guarantee. The capability-matching estimator also lacks sufficient specification and calibration evidence to justify treating its scores as success probabilities.

  2. BRAID does not establish the claimed coordination requirement. It bundles separately answerable questions and scores them independently. Without a question-wise single-agent baseline and stronger workflow comparators, the gains may reflect task separation rather than inter-agent coordination or reliable fault localization.

Questions for authors
  1. How can the non-degradation claim be reconciled with Figure 8? How often do accepted repairs improve or worsen final answers?
  2. How is the rubric estimator implemented, and how accurately do its scores predict agent success and contract violations?
  3. How does INFLOWOP compare with a single agent solving each bundled question separately under matched token budgets?
  4. What makes COALESCE’s pruning bound admissible when merging re-scores existing subtasks and can reduce their cost?
3 comments
AuthorsOpenReview ↗

Author Response (1/2): Thank you for your valuable review and insightful feedback

Thank you for your careful review and constructive feedback, and for recognizing that our scoped repair avoids unnecessary workflow re-execution, and that our ablations distinguish the gains from construction, in-flow repair, and evolving profiles. We hope to further address your remaining concerns below.

W1+Q1: Non-degradation guarantee and Figure 8

Thank you for your insightful question. We hope to respectfully clarify the guarantee and add the per-task breakdown below.

1. Scope of the guarantee (Q1 "How can the non-degradation claim be reconciled with Figure 8?"). The guarantee concerns the label-free score that guides both construction and repair, together with the progress that execution has already achieved. A repair is kept only if it scores strictly higher than the original on the same detector scale. So a tie or an unscoreable repair keeps the original. Every committed subtask therefore scores at least as high as its constructed realization, and execution resumes over every result already computed (§3.2). Final-answer correctness is a separate quantity, which is why every result in §5 is end-to-end task accuracy. In our revision, we will state this scope clearly in §3.2.

2. Figure 8 (W1 "contract judgments can disagree with downstream correctness"). Fig. 8 (§D.5) illustrates this distinction, as its title states: "The Worth of A Step Is Settled Downstream". Scored against its own condition alone, a step that helps the flow reach the correct answer can be recorded as the step that failed. The figure is therefore a diagnostic on scoring a step in isolation, not a trace of a committed repair, and we will state this clearly in its caption.

3. How often accepted repairs improve or worsen final answers (Q1 "How often do accepted repairs improve or worsen final answers?"). In aggregate, in-flow optimization raises Qwen3.5-9B from 22.21% to 23.80% (Tab. 2), net of every task it improves or worsens. We further add the per-task breakdown over the 16 arms of Tab. 1 with GPT-5.6-luna: a repair is accepted on 826 tasks, and on them the final score improves on 85.23% (Δ ≥ 0, where Δ is the task's score with in-flow optimization minus its score with COALESCE alone) and worsens on 14.77% (Δ < 0). Accepted repairs thus improve the final score on the large majority of the tasks they are applied to.

W1+Q2: Estimator implementation and calibration

Thank you for your thoughtful question. We hope to respectfully provide the full specification, and clarify how the scores are used.

1. Implementation (Q2 "How is the rubric estimator implemented"). The rubric estimator scores how well a supplied text satisfies a demanded one (§3.1), without any LLM call:

  • Encoding. A frozen sentence encoder (all-MiniLM-L6-v2) embeds both texts, and an increasing function maps their cosine similarity to p^∈[0,1]\hat p \in [0, 1], so the score is deterministic and requires no sampling or training.
  • Supply and demand. The supplied text is the agent's card (Agent Card box, p.19), which states its declared competence, together with its self-evolving empirical profile when profiles are enabled (Fig. 5). The demanded text is what the subtask requires. Where a subtask declares several conditions, each is scored on its own, and the weakest condition sets the score.
  • Profile. The profile term is computed as a kernel-weighted average over the agent's accumulated records, and it is combined with the card term by a weight that grows with the number of records. So the profile takes over only as evidence accumulates.

In our revision, we will add this specification to §D.

2. Calibration (W1 "calibration evidence to justify treating its scores as success probabilities", Q2 "how accurately do its scores predict agent success and contract violations"). Every decision INFLOWOP makes from the scores depends only on how they compare with one another or with a fixed bar on the same scale: each subtask is bound to its lowest-cost agent (Alg. 3), agent creation is triggered when no agent clears the competence bar Cˉ\bar C (Eq. 1), a condition is unmet once it reaches the unmet bar uˉ\bar u (Eq. 3), re-assignment takes the next-best agent (Eq. 20), and a repair is kept only if it scores strictly higher than the original. Rescaling the scores and bars in any way that preserves their order therefore leaves every decision unchanged. So none of these decisions requires the scores to be calibrated success probabilities. The scores enter as values only where Eq. 2 sums reliability with latency and creation, and there p^\hat p serves as a surrogate objective (§3), validated end to end. Given the same agent cards, profiles, and atom demands, the rubric estimator is higher than an LLM estimator by +3.27 on Qwen3.5-4B and +3.85 on Qwen3.5-9B (Fig. 3), and letting created agents join its cost matrix adds +3.15 and +2.55 (Fig. 4). Where its scores keep a repair, the final score improves (Δ ≥ 0) on 85.23% of tasks (W1+Q1).

AuthorsOpenReview ↗

Author Response (2/2): Thank you for your valuable review and insightful feedback

W2+Q3: Coordination in BRAID and workflow comparators

Thank you for your thoughtful comments. We hope to respectfully clarify what BRAID requires, and compare with a single agent per question and two established workflow methods below.

1. What BRAID requires (W2 "It bundles separately answerable questions and scores them independently"). Each BRAID task provides more sources than its questions need, without stating which source serves which question (§C.2). In a Distributed task, each of the kk questions has its own gold source among 2k−12k-1 sources, so sources have to be assigned to questions jointly, since the source that answers one question answers no other. In an Anchored task, all kk questions are answered from one gold source among kk sources, so every part depends on first locating that source. Answering a task thus requires dividing it and coordinating the routing to its answers, and scoring each question on its own keeps this requirement intact. Fig. 7 (§C.1) shows its effect: on the original datasets, a single agent is more accurate than a workflow an LLM constructs (54.37% vs. 42.35%), while after BRAID composition the single agent drops to 10.82% and the workflow reaches 48.99%.

2. Gains beyond task separation (W2 "the gains may reflect task separation"). The greedy-search baseline already separates each task across agents, yet it reaches 15.49% against the single agent's 17.38% (Tab. 2). Deciding the division by cost reverses this: COALESCE reaches 22.21%, and INFLOWOP 25.13%. Since greedy-search already separates the task, the +9.64 of INFLOWOP over it comes from how the task is divided and repaired, as §5.2 states: "Gains come not from dividing the task, but from how it is divided".

3. A single agent per question (Q3 "a single agent solving each bundled question separately under matched token budgets"). Solving each question separately is one coalescing that COALESCE prices under the same objective as every other coalescing (Eq. 2). Such a split still leaves each question to locate its own gold source among all the sources of its task, which is the routing that BRAID composition introduces. Following your suggestion, we further add a single agent solving each bundled question separately under matched token budgets. We show overall accuracy (%) with Qwen3.5-9B below:

MethodMean over 16 arms
Single agent, one run per task (Tab. 2)17.38
Single agent, one run per question19.39
INFLOWOP (Tab. 2)25.13

This baseline is stronger than the Single Agent in Tab. 2, yet INFLOWOP is higher than it by +5.74, and in each of the 16 arms of Tab. 1.

4. Stronger workflow comparators (W2 "stronger workflow comparators"). We add two established workflow methods, AFlow [1] and DyFlow [2]. As released, both are text-in/text-out and cannot open BRAID's documents and images, so we run both with Qwen3.5-9B and the same tools, skills, sandbox, budget, and run limits as our INFLOWOP, on five datasets of Tab. 1 with document, image, and text inputs. Mean accuracy (%) over each dataset's arms below shows that INFLOWOP is the highest on every dataset, by +4.48 to +12.67 over the stronger of the two methods:

MethodMMLongBench-DocDocFinQAChartMuseumOlympiadBench-Math (Text)OlympiadBench-Physics (Text)
AFlow10.3316.1026.9133.144.20
DyFlow8.5018.1131.3437.305.63
greedy-search8.8513.4617.6337.674.53
INFLOWOP14.8125.2041.2849.9710.88

Q4: Admissibility of the pruning bound

Thank you for this insightful question. COALESCE prunes a partial coalescing once its lower bound reaches the cost of the best complete coalescing found (§D.1, Alg. 3). The bound is the cost committed so far plus the least cost the unplaced atoms can still incur: each unplaced atom contributes its cheapest single-atom cost, and each distinct new agent they require contributes one creation penalty. So atoms that share a created agent are never charged twice. This bound is admissible whenever a merge does not lower the cost of the subtask it forms (Q4 "merging re-scores existing subtasks and can reduce their cost"). In our revision, we will state this clearly in §3.1 and §D.1.

References

[1] J. Zhang et al. AFlow: Automating Agentic Workflow Generation. 2025.

[2] Y. Wang et al. DyFlow: Dynamic Workflow Framework for Agentic Reasoning. 2025.

Thank you for your valuable review and constructive feedback. We sincerely hope our response and clarifications above can meaningfully address your remaining concerns, and we will make sure to apply them to our revisions to further enhance our paper quality.

Thank you for the additional experiments. I’ll keep my rating at 2, since I think the failure attribution of this method is too coarse to guide repairs reliably.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.