- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes OpWeave for automated agentic workflow design. The framework expands its operator space through verified Operators and reusable Pattern cards. It uses execution traces and hierarchical attribution to guide edits at the Block, Path, or multi-block level. Short-term memory supports the current search, and long-term memory stores capabilities and priors for later tasks. Experiments cover six benchmarks in reading comprehension, code generation, and mathematical reasoning, with a matched executor and search budget. The results show competitive performance and substantial token savings from search-free transfer. The continual-accumulation experiment shows clearer benefits for reuse and search efficiency than for accuracy.
Critical strengths
The main contribution is allowing the workflow language to expand and persist across tasks. Verified Operators add capabilities, and Pattern cards store useful local compositions. This gives the system a way to reuse both individual operations and workflow structures. The optimization design is also well motivated. Execution traces guide the scope of an edit, from a Block to an executed Path or a larger region. Results on DROP, HumanEval, and MATH suggest that these localized edits can improve search under the same evaluation budget.
The evaluation considers search cost as well as final performance. Using a matched executor and search budget makes the comparisons easier to interpret. Search-free transfer is particularly useful in practice: it produces competitive workflows at a small fraction of the token cost of full search. I also appreciate the mixed results. Continual accumulation reduces search cost but does not consistently improve accuracy, and the value of same-type priors varies across tasks. These findings help identify the actual benefits of memory rather than presenting reuse as uniformly helpful.
Critical weaknesses
The continual-learning evidence is still limited. The accumulation experiment uses one task order and apparently one run per step. Its average test score is slightly below the corresponding cold runs. The result therefore supports reuse and efficiency more clearly than improved accuracy. The ablations also leave some component effects unclear. Table 2 adds Briefs, Operators, Patterns, and memory in sequence, despite non-monotonic effects across tasks. Table 3 covers only three of the six benchmarks. Limited repetitions further complicate the interpretation of small differences. The transfer discussion should also separate cross-task memory from the “all tasks + feedback” setting, which includes previous target-task optimization feedback.
The experiments use closed-form benchmarks and one executor model. This leaves the broader motivation for general agentic workflow design only partly tested, especially for long-horizon tool use and open-ended environments. The design phase also combines a strong model, retrieval, and verification. The comparisons do not clearly isolate the benefit of typed Operator/Pattern registration from this additional assistance. A control with the same model, retrieved evidence, and budget would help. Finally, fewer search rounds may not mean lower end-to-end cost. Attribution and verification add overhead, so token use, model calls, and wall-clock latency should be reported separately.
Questions for authors
Q1. Are the continual-accumulation results stable across task orders and random seeds? In particular, does the reported 0.84× average search cost hold across runs, and how much does average accuracy vary? Please also separate transfer from other tasks from reuse of previous target-task feedback. This distinction matters for interpreting the cross-task generalization claim.
Q2. Could you add a leave-one-component-out design ablation and a control with the same design model, retrieved evidence, and token budget, but without explicit Operator/Pattern registration? Do Path restriction, stall escalation, and short-term memory show similar benefits on HotpotQA, MBPP, and GSM8K? End-to-end token use, model calls, and latency would also help assess the cost of these mechanisms.
Response to Reviewer dBrB
Dear Reviewer dBrB, thank you for the precise questions, and for noting that the pipeline is coherent and that the shared-initialization comparison and the distinction between cross-task and cross-type memory are useful. We agree with each qualification you ask for in the second weakness and have narrowed the corresponding claims. Below we answer Q1–Q4 together with the weaknesses they relate to, and name the table that carries each number.
1. Q1 and Weakness 1 (first part): does Operator expansion outperform a fixed library?
Yes, when Operators come with the Pattern cards that compose them: against a fixed-library control, the full design scores 76.5 against 71.5 on the six-benchmark test average (Table R1, General Response G1). We agree with your premise that this is not a gain in expressive power. The admitted Operators are specialized prompt modules, and a generic operator can express them: a registered Operator is a stored prompt executed through the generic
implemented_skilloperator. The control shows that the Designer does not recover that behavior without the registry.The control. The "w/o O and P" row of Table R1 (General Response G1) is our fixed-library control. Its run differs from the Full run in two flags only, which remove the admitted Operators and the Pattern cards from the registry section of the Designer prompt. The Designer (GPT-4o-mini, temperature 0.2), the Task Brief, the memory file, and the registry file are identical, and the default library remains available. Recalled memory does not bring the withheld elements back: the memory section that enters the prompt contains no Operator or Pattern name. The Designer is free to write any specialized prompt into the default operators. It scores 71.5 against 76.5 for the full design (MATH 32.2 against 50.2; MBPP 68.2 against 77.7).
Identical specialized prompts. Pasting the registered prompt text verbatim into generic operators is the Full system itself, since a registered Operator is a stored prompt run through
implemented_skill. The control therefore withholds the registered prompts and leaves the Designer free to write its own. What registration adds beyond the prompt text is the declared interface, the acceptance fixtures, and persistence across tasks.Independent value of Operators. Your concern is correct for Operators taken alone. Relative to the control, Pattern cards alone add 2.0 points, Operators alone add 0.1, and the two together add 5.0. We therefore describe the contribution as expansion of the workflow language, meaning Operators plus the Pattern cards that compose them, and no longer as Operator expansion by itself.
Rationale for prompt-backed Operators. This is a deliberate restriction. An admitted Operator can only issue a model call under its declared schema. It cannot call the web, a shell, or code execution, and it adds no tool beyond those of the default library. We chose this for two reasons. First, admission stays cheap and checkable: a prompt-backed Operator has no side effects, so a schema check and 1 to 4 fixtures are enough to enforce its contract (Table R9). Second, the comparison stays clean: the full design and the fixed-library control have identical tool access, so the 5.0-point difference in Table R1 (General Response G1) cannot come from added external capability. The same holds against the baselines: the operators of AFlow and A²Flow have the same tool access as ours, so the margin over them (Table 1; Table R5c, General Response G5) does not come from extra tools either. We acknowledge that this design also limits what Operators can do. What prompt-backed Operators offer in practice is portability. The workflows that contain them run unchanged on Qwen3.5-9B and Gemma 4 12B and stay above single-call prompting (Tables R6b–R6c, General Response G6). Admitting tool- or code-backed Operators would need stronger verification, and we leave it to future work.