- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes FlowRight, a framework for optimizing multiple roles involved in constructing and executing LLM workflows. It also introduces DataWright, which constructs harder evaluation tasks through question bundling and distractor inputs. The experiments demonstrate the framework is potentially useful. However, the paper does not clearly show that tasks created by DataWright require workflow coordination, or that coevolution itself drives the performance gains.
Critical strengths
- The framework accommodates workflow generation, component invention, and downstream execution within a common training setup.
- Beyond the training distribution, improvements also appear in the aggregate results for unseen task configurations and domains, providing encouraging evidence of generalization.
Critical weaknesses
- DataWright does not establish that its tasks require learned workflow coordination. Shared and Paired tasks combine independent questions, while Decoy tasks mainly require finding the relevant input. A fixed workflow that solves each question separately may perform similarly. The paper should compare against this baseline and a single agent with comparable resources to show that the gains come from learned coordination.
- The paper does not specify which roles are trained together or how they are updated. It is therefore unclear whether the reported gains come from mutual adaptation or simply from training more components. A useful control would train the roles independently and then combine them under the same total training budget.
- Section 3.5 does not explain where the successful examples used for test time distillation come from, when the resulting prior is fixed, or whether it changes across test tasks. These details are needed to interpret the reported gains. The authors should also clarify why Table 3 lists training questions for DocFinQA, which Section D.5 describes as used only for evaluation.
- Section D.4 uses token-level F1 for open-ended answers in Shared and Paired tasks, but exact match in Decoy tasks. Scores across these strategies are therefore not directly comparable. Table 3 also lists 1,065 test questions for InfoChartQA Visual at level 3 and 1,735 at level 5 under Paired, making it unclear whether the two levels use the same question pool. The authors should report results with consistent scoring and matched questions.
Questions for authors
See weaknesses.
7 comments
Author Response (2/2): Thank you for your valuable review and insightful feedback
W3: Test-time distillation and DocFinQA
Thank you for your detailed comments.
1. Test-time distillation: Internal distillation draws reusable knowledge from the system's own trajectories on earlier tasks of the same test stream, and external distillation uses a stronger teacher as the distiller (§3.5). A successful example is a workflow that executes to a final answer, so the distiller never receives a grade, a label, or a reference answer. The prior is updated every tasks ( by default): the first tasks use the base prior, and each later block uses the prior distilled from the blocks before it, fixed within that block. The prior thus changes across test tasks by design, online and without gradients, and is never fitted to the task being answered. In our revision, we will state , this schedule, and the source of the trajectories in §3.5.
2. DocFinQA: We respectfully clarify that DocFinQA serves only for evaluation: RL trains and validates only on the Paired ℓ=3 arms of InfoChartQA-Visual, TACO, SlideVQA-Doc, and LongDocURL (Tab. 1). DataWright also builds a training split to benefit future research since DocFinQA's original dataset has a training split. But no training, distillation, or pool growth uses it, and we will mark this clearly in Tab. 3 in our revision.
W4: Consistent scoring and matched questions
Thank you for your careful comments.
1. Consistent scoring (W4 "report results with consistent scoring"). Every comparison is within an arm, where all methods share one metric, so the metric cannot favor a method. We nevertheless re-score the Decoy text answers with the partial credit of Eq. 9 that Shared and Paired already use, on the six Decoy arms with text answers (DocFinQA's numeric answers receive no partial credit). We show the gain (%) over the untrained FloWright with Qwen3.5-4B below:
| Mode | Exact match (paper) | Consistent scoring |
|---|---|---|
| (a) single-role | +3.61 | +4.43 |
| (b) agent-skill | +4.93 | +6.05 |
| (c) upstream-downstream | +4.18 | +5.43 |
| (d) multi-agent | +5.32 | +6.89 |
Every mode stays above the untrained workflow in all six arms, and every gain grows. We will add these results to §G.1 and note the Decoy scoring in §4.
2. Matched questions (W4 "making it unclear whether the two levels use the same question pool"). Both levels draw from the same InfoChartQA-Visual test pool: 1,048 of the 1,065 ℓ=3 questions also appear at ℓ=5. The two levels hold a similar number of tasks (355 and 347) with three and five questions each, so ℓ=5 draws more questions (1,735) from that pool. On the 194 questions evaluated at both levels, the gain (%) over the single agent with tools and skills below shows that FloWright's lead grows at the higher level, with Qwen3.5-4B:
| Method | Gain over the single agent, ℓ=3 | Gain over the single agent, ℓ=5 |
|---|---|---|
| FloWright, untrained | +14.07 | +19.59 |
| FloWright, (d) multi-agent | +22.44 | +25.46 |
In our revision, we will add these matched results to §G.1 and clarify the question counts of this dataset in §D.5.
References
[1] J. Zhang et al. AFlow: Automating Agentic Workflow Generation. 2025.
Thank you for your valuable review and constructive feedback. We sincerely hope our response and clarifications above can meaningfully address your remaining concerns, and we will make sure to apply them to our revisions to further enhance our paper quality.
Thank you for your rebuttal. I have carefully read your responses and appreciate the clarifications. I will maintain my current score. Best of luck!
Author Response (1/2): Thank you for your valuable review and insightful feedback
Thank you for your careful review and constructive feedback, and for recognizing our training setup for workflow generation, component invention, and downstream execution, and our encouraging evidence of generalization. We hope to further address your remaining concerns below.
W1: Workflow coordination in DataWright tasks
Thank you for your insightful comment. We hope to respectfully clarify that DataWright constructs tasks beyond a single agent's ability (App. C), while learned coordination comes from FloWright's training. Below, we show results against a fixed workflow and a single agent with comparable resources.
1. Single agent with comparable resources. Tab. 1 already includes a single agent with the same tools, skills, sandbox, and run limits, which receives on each test sample the agent turns the compared workflow used on that sample. The untrained FloWright outperforms it by +11.39 at 4B and +16.75 at 9B overall (Tab. 1). On MMLongBench-Doc hardened by DataWright, the same workflow reaches 51.61% against the single agent's 12.33% (App. C). We will add these budget matching details to §F.2.
2. Fixed per-question workflow (W1 "A fixed workflow that solves each question separately may perform similarly"). The untrained FloWright, the reference of every gain in Tab. 1, already splits a bundled task into steps (9.74 per workflow on average at 4B) and aggregates their answers (Fig. 11). Over it, training improves accuracy by up to +5.03% at 4B (Tab. 1). To further validate, we additionally compare with AFlow [1], which searches one fixed workflow per dataset on labeled training tasks and applies it to every test task, with Qwen3.5-9B and our tools, skills, sandbox, budget, and run limits. On the 23 arms with training splits, the untrained FloWright is higher than AFlow in every arm (42.51% vs. 34.61% mean accuracy).
3. Gains with nothing to decompose. Each Decoy task holds a single question among decoy inputs (§D.1). On all 8 Decoy arms, the untrained FloWright outperforms the single agent with tools and skills by 6.96 at 4B and 12.03 at 9B on average, and every 4B mode improves all 8 arms further, by about as much as on Shared and Paired (§G.1). The gains therefore come from learned workflow construction beyond per-question decomposition.
W2: Roles trained together and their updates
Thank you for your thoughtful comment.
1. Roles and updates (W2 "which roles are trained together or how they are updated"). Mode (b) trains the Generator and the Inventor skill, (c) the Generator and the downstream agent, and (d) all three. Each role is its own policy with no shared weights, and co-evolving roles learn from one shared set of rollouts, each taking one optimizer step per training step. In our revision, we will state this clearly in §3.4.
2. Roles trained independently, then combined (W2 "train the roles independently and then combine them under the same total training budget"). Following your suggestion, we combine the roles from their own single-role training (Fig. 14) into one system. GPU-hours count one role's GPUs as one unit, and one single-role run takes 50.04 GPU-hours and 8,000 rollouts. We show overall accuracy (%) with Qwen3.5-4B below:
With 1.08-1.13 times the GPU-hours and one shared set of rollouts, co-evolution is 3.22-6.48 points higher, while the separately trained roles fall below even the untrained workflow (Tab. 1). The gain thus comes from the roles adapting to each other during co-evolution rather than from training more roles.