- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Trace2Workflow, which converts agent interaction traces into reusable executable workflows in three stages: (1) extract parameterized local programs from successful and failed traces and keep only candidates with no verified failure and sufficient positive evidence under each local goal's completion check; (2) compile the task-level composition of a successful trace into a workflow W = (x, V, β, g, Φ) with argument bindings, an applicability guard, and observable completion checks; (3) on a new task, execute a matching workflow without model calls and DEFER to the LLM from the current state if any check fails. With Qwen3.5-27B as the backbone and ReAct, CodeAct, and Skill-DisCo as baselines, it reports success rates of 97.86/98.51 on ALFWorld Seen/Unseen with 0.879/0.858 LLM turns per task, and 83.33/70.00 on AppWorld test-normal/test-challenge.
Critical strengths
- Clear idea with a sensible safety mechanism. Compiling the composition itself, rather than only local skills, directly targets repeated task-level planning, and the guard + effect check + DEFER design returns control to the LLM from the current state without resetting the environment or budget, which is more practical than all-or-nothing replay. The ablations (Table 3) and the reuse-granularity analysis make each component's role visible.
Critical weaknesses
- The strongest evidence comes from a setting where reuse is nearly trivial, while the evidence on the harder benchmark is thin. ALFWorld has six fixed task templates whose action sequences differ mainly in objects and receptacles, so a parameterized workflow from one successful trace is expected to cover most test tasks; achieving zero model calls on 96 (Seen) and 99 (Unseen) tasks therefore says more about the benchmark than about generalization. The Unseen split changes room layouts, not task types. On AppWorld, where API interactions are diverse, only 4/60 and 2/60 tasks are completed without model calls, and the gains over CodeAct are 50 vs. 47 and 42 vs. 37 tasks, i.e., 3–5 tasks per split, on 60-task subsets whose sampling procedure is not described.
- The baseline comparison on AppWorld is not interpretable. With the same Qwen3.5-27B backbone, ReAct solves 7/60 and 0/60 AppWorld tasks while CodeAct solves 47/60 and 37/60. ReAct also has the highest average turns (32.2 and 34.4), suggesting most episodes exhaust the action budget, but the budget value is not given. Appendix B.1 provides only a combined "Agent and CodeAct Prompt" describing an interactive Python environment, and no ReAct-specific prompt or action format, so it is unclear how ReAct and CodeAct actually differ in this implementation. Without this, the claim of outperforming ReAct on AppWorld cannot be assessed.
Questions for authors
- (Clarification + additional experiment) How were the 60 AppWorld tasks per split sampled? Can you report results on the full test-normal / test-challenge splits?
- (Clarification) What prompt, action format, and step budget does ReAct use, and why does it solve 0/60 on test-challenge?
- (Clarification) On ALFWorld, what are coverage and zero-LM rates per task type? Are some types never covered by a workflow?
- (Clarification) The guard and completion checks are written by the LLM at compile time from a single trace (Appendix B.4). How often does a guard match a task to which the workflow does not actually apply?
3 comments
Response to W2 and Q2: ReAct Baseline
We thank the reviewer for the comments on the ReAct baseline. We provide more details about its implementation and the performance differences.
1. ReAct and CodeAct Implementation
Both ReAct and CodeAct use the same Qwen3.5-27B backbone and task environment. Their main difference is the action format.
In our implementation, ReAct generates one JSON action per turn, including "thought", "app", "api", and "arguments". Each action calls only one public API. In contrast, CodeAct generates executable Python code. It can use variables, loops, and multiple API calls within a single model turn.
We will provide the separate prompts, action formats, step budgets, and stopping rules in the revised appendix to make the comparison clearer.
2. ReAct Performance on AppWorld
AppWorld contains complex tasks that require multiple API calls, data processing, and repeated operations. For ReAct, each API call requires a separate model decision. In contrast, CodeAct can combine several operations into one Python program.
This difference has a large impact on complex tasks. ReAct spends more model turns on low-level API operations, while CodeAct can complete the same operations with fewer model interactions. In addition, invalid action outputs also consume model turns in our ReAct implementation.
These differences lead to ReAct's lower performance, especially on test-challenge. The reported 0/60 result reflects the difficulty of completing all task requirements under this execution setting.
3. Baseline Comparison and Fairness
Our main conclusion does not depend on ReAct's performance. Trace2Workflow also outperforms the stronger CodeAct baseline. On AppWorld, success improves from 47/60 to 50/60 on test-normal and from 37/60 to 42/60 on test-challenge. Average LLM turns decrease from 20.517 to 11.017 and from 26.500 to 14.950, respectively.
Furthermore, Local Programs Only and Trace2Workflow use the same local-function library and training evidence. Their main difference is whether task-level workflows are reused. This comparison directly supports the benefit of workflow-level reuse.
We will clarify the baseline implementation details and emphasize these comparisons in the revised paper.
Response to Q1 and Q4: AppWorld Evaluation and Guard Reliability
We thank the reviewer for these questions. We clarify the AppWorld sampling method and provide further analysis of workflow applicability.
1. AppWorld Task Sampling and Full Evaluation (Q1)
Our original evaluation uses template-level random sampling. For each split, we randomly select 20 task templates without replacement and include all three instances of each template. This gives 60 tasks per split. We use a fixed random seed of 42. The sampling process uses only task IDs and does not consider model performance or workflow coverage.
The full test-normal and test-challenge splits contain 168 and 417 tasks, respectively. To further evaluate generalization, we conduct a full-split comparison between CodeAct and Trace2Workflow using the same Qwen3.5-27B backbone and evaluation settings.
| Dataset | Method | Success | SR (%) ↑ | Turns ↓ | Zero-LM ↑ |
|---|---|---|---|---|---|
| test-normal (168) | CodeAct | 103/168 | 61.31 | 21.300 | 0 |
| Trace2Workflow | 118/168 | 70.24 | 17.800 | 5 | |
| test-challenge (417) | CodeAct | 229/417 | 54.92 | 27.200 | 0 |
| Trace2Workflow | 283/417 | 67.87 | 22.100 | 4 |
Trace2Workflow improves the success rate by 8.93 and 12.95 percentage points on test-normal and test-challenge, respectively. It also reduces average LLM turns by 16.43% and 18.75%. These results show that workflow reuse remains effective on the full AppWorld splits.
We will provide the sampling script and task IDs for reproducibility.
2. Guard Applicability and Reliability (Q4)
During offline compilation, the LLM generates request patterns and applicability guards from successful traces. At runtime, the matcher checks input bindings and guards before execution. Only a unique valid match is accepted.
We also verify API responses and observable effects during execution. If a check fails, the workflow returns DEFER, and the LLM continues from the current state.
We evaluate guard reliability by checking whether each accepted workflow matches the task requirements. We distinguish incorrect matches from execution failures. The results are shown below.
| Metric | test-normal (168) | test-challenge (417) |
|---|---|---|
| Accepted Matches | 22 | 30 |
| False Matches | 5 | 11 |
| False-Match Rate (%) | 22.73 | 36.67 |
The false-match rate is higher on test-challenge, showing the difficulty of matching workflows to diverse tasks. These results also show the limitations of generated guards. Runtime checks provide an additional way to stop execution when observable conditions fail. The LLM can then continue from the current state.
We will include this analysis and clarify the role of guards, runtime checks, and fallback in the revised paper.
Response to W1 and Q3: Motivation and Workf
We thank the reviewer for this comment. Our main goal is to avoid repeated planning for similar tasks. ALFWorld fits this goal well. Its tasks share similar procedures but differ in objects, locations, and environment states. Compared with Local Programs Only, our method reduces average LLM turns from 2.806 to 0.858 on ALFWorld Unseen. This shows the benefit of reusing task-level workflows.
1. Per-Task-Type Analysis on ALFWorld
We further analyze workflow coverage and Zero-LM across the six ALFWorld task types. Coverage measures the percentage of tasks that execute a workflow. Zero-LM measures the percentage of tasks completed without LLM calls.
On Unseen, workflows directly complete all 99 tasks from the first four types. This shows that workflow reuse is effective for repeated task structures, but its coverage depends on the learned workflow library.
2. Generalization on AppWorld
AppWorld is more challenging because its tasks are more diverse and share fewer common workflows. As a result, fewer tasks can be completed without LLM calls. However, local programs and partial workflow reuse can still reduce repeated planning. Compared with Local Programs Only, our method reduces average LLM turns from 18.350 to 11.017 (40.0%) on test-normal and from 23.217 to 14.950 (35.6%) on test-challenge.
Our goal is not to achieve Zero-LM on every task, but to reduce repeated planning whenever possible. We will clarify that our method focuses on reusing experience across similar tasks, rather than solving arbitrary unseen task types.