- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper organizes the execution records of similar tasks into reusable text procedures. When a new task arrives, the corresponding procedure is selected, the current task information is filled in, and then it is handed over to the agent for execution. The authors compared the performance and execution costs of tasks on multiple environments and models, and also tested whether the procedures organized by the strong model could help the small model.
Critical strengths
S1. The method has shown improvements in multiple environments and models, and some settings have also reduced the number of execution steps.
S2. The procedures organized by a strong model can directly assist a small model, and in most test settings, they are more effective than the procedures organized by the small model itself. This indicates that these experiences can be reused across models without the need for retraining the executing model.
S3. The comparison of equal-length irrelevant prompts is useful, indicating that the benefits are not solely due to longer prompts. The authors also included a simple but effective control that directly retrieves successful records.
Critical weaknesses
W1. I think the added contribution of the method is rather limited. My initial reaction was, isn't this just a workflow? "AWM" can already generate reusable text workflows from execution experience, and "Agentic Plan Caching" also saves and reuses plan templates. This article emphasizes preparing a complete procedure in advance for each type of task, but it does not directly compare this with similar methods. It is not clear what benefits this approach brings over the existing workflow reuse.
W2. The appendix provides estimates for compilation costs and break-even points, but it does not clarify whether collecting execution records, retrying failed tasks, and verifying modifications are all included. If these costs are not accounted for, it is unclear how many tasks would be needed to recover the cost when establishing a procedure library from scratch.
W3. Figure 1 compares each execution record with the most similar one, but finding similar examples does not mean that the entire type of task can share a single procedure. Even within the same type of task, even if there are several different approaches, each record may still find very similar neighbors. It is not clear whether preparing a procedure in advance is still more effective than per-task orchestration when the differences in these approaches are significant.
Questions for authors
See weaknesses.
Can you compare the small model's performance when using the original successful execution records of the strong model and when using the procedure derived from these records, under similar context budgets? I want to see if converting these trajectories into a workflow provides any additional value for the small model, or if directly providing the strong model’s successful execution trajectories can achieve a similar effect.
6 comments
Response to Reviewer Z9CX – Part II
W2: Thank you for asking us to clarify the scope of the cost accounting. Appendix C.9 reports the incremental cost of the trajectory-to-prefab compilation calls. The corresponding break-even estimate therefore concerns amortizing this compilation overhead. FlatSkill supports an iterative reuse cycle: an agent starts with ReAct+Skills, compiles its collected execution experience into prefabs, and can subsequently compile updated prefabs from new prefab-augmented trajectories. Section 4.6 already evaluates this cycle: a second round of compilation improves ScienceWorld reward from 86.3 to 89.0 on the seen split and from 79.4 to 84.4 on the unseen split. This allows accumulated experience to improve a reusable task-level composition across rounds. Other skill orchestration Methods such as GraSP use collected trajectories only to guide orchestration for each new task, and FlatSkill retains the resulting composition itself for reuse across subsequent instances.
W3: Thank you for raising this point. Tasks of the same type often share a common sequence of key actions, even when their complete executions differ. FlatSkill extracts these execution backbones from multiple trajectories and compiles the shared steps into a slotted procedure. We then compare successful and failed trajectories to propose outcome-verified rules that explain what the agent should do when particular conditions arise. Together with the branches retained during procedure compilation, these rules help the prefab handle differences between specific tasks. So sharing one prefab does not require every instance to follow an identical execution path. Figure 1 illustrates recurring patterns, while compilation considers multiple trajectories to capture both shared steps and useful conditional behavior. On-the-fly orchestration, like GraSP also uses past trajectories as memory to guide orchestration for the current task. FlatSkill uses execution experience to learn the common procedure and conditional guidance ahead of time, allowing subsequent tasks to reuse this composition while adapting their actions to the environment.
Q: Thank you for suggesting this comparison. We conducted two additional experiments with Qwen3-8B:
-
The small model directly uses the strong model’s trajectories. We provide Qwen3-8B with up to eight original successful execution records from DeepSeek, retrieved by task-text similarity from the candidate trajectories originally used for procedure compilation for the same task type. We also conduct a comparison under similar context budgets, selecting the single most similar record and match its context budget to the corresponding DeepSeek-compiled prefab. This record contains the original task description, reasoning, actions, and observations, with the skill documents removed from the original system message. When it exceeds the budget, we retain its beginning and end and remove the middle.
-
The small model compiles the strong model’s trajectories. We ask Qwen3-8B to compile prefabs from DeepSeek’s trajectories and then execute tasks using these prefabs. For comparison, we also include prefabs compiled by Qwen3-8B from its own trajectories and prefabs compiled by DeepSeek from DeepSeek’s trajectories.
| Input to Qwen3-8B | ALFWorld Seen | ALFWorld Unseen | WebShop Seen | ScienceWorld Seen | ScienceWorld Unseen | MedCalc Seen |
|---|---|---|---|---|---|---|
| DeepSeek trajectories, full | 79.7 / 13.9 | 82.8 / 13.5 | 25.9 / 20.9 | 75.9 / 15.0 | 67.8 / 17.1 | 83.5 / 2.6 |
| DeepSeek trajectory, matched token budget | 28.6 / 25.3 | 35.8 / 24.7 | 5.3 / 27.6 | 60.6 / 18.5 | 49.5 / 20.4 | 42.8 / 3.5 |
| Prefab: DeepSeek trajectories → Qwen3-8B compilation | 74.3 / 14.1 | 76.9 / 15.0 | 24.0 / 21.0 | 72.9 / 15.2 | 68.6 / 16.3 | 65.2 / 6.5 |
| Prefab: Qwen3-8B trajectories → Qwen3-8B compilation | 81.4 / 12.9 | 79.9 / 13.7 | 38.9 / 11.6 | 76.8 / 15.1 | 71.3 / 16.6 | 85.6 / 2.0 |
| Prefab: DeepSeek trajectories → DeepSeek compilation | 86.4 / 12.0 | 91.8 / 11.7 | 41.7 / 15.3 | 77.0 / 14.4 | 73.0 / 16.0 | 84.2 / 2.0 |
The additional experiments show that providing the strong model’s trajectories to Qwen3-8B, either to guide execution directly or to compile prefabs, does not improve overall reward over the prefabs compiled by Qwen3-8B from its own trajectories already reported in the main paper. This holds both under comparable context budgets and when up to eight original successful execution records are provided. The DeepSeek-compiled prefabs already evaluated in the paper achieve higher reward with fewer interaction steps in all six settings than either directly supplying these records or asking Qwen3-8B to compile them. These results show that the cross-model benefit comes from transferring the strong model’s compiled prefabs, rather than providing its trajectories alone.
Dear Reviewer, we sincerely appreciate your time and effort in reviewing our manuscript and offering valuable suggestions. As the author-reviewer discussion phase is drawing to a close, we would like to confirm whether our responses have effectively addressed your concerns. We provided detailed responses to your concerns a few days ago, and we hope they have adequately addressed your issues. If you require further clarification or have any additional concerns, please do not hesitate to contact us. We are more than willing to continue our communication with you. Best regards.
Response to Reviewer Z9CX – Part I
Thank you for your careful review and for recognizing FlatSkill’s improvements across environments, cross-model reuse, and controlled ablations. We address your concerns below.
W1: Thank you for highlighting the connections to AWM and Agentic Plan Caching. We appreciate the opportunity to clarify FlatSkill’s contribution and provide direct comparisons with these methods.Our core comparison GraSP, also builds on skill-augmented trajectories, but retrieves and composes skills on the fly for each task instance. FlatSkill consolidates the realized action compositions across multiple instances of the same task type into a prefab before inference. A task type may require multiple skills; FlatSkill makes their realized composition reusable as a unit through routing and slot binding, avoiding repeated skill retrieval and composition for subsequent instances.
FlatSkill’s central contribution is to make task-type-level skill composition a reusable unit. It consolidates the action compositions realized in multiple skill-augmented trajectories into a prefab that can serve subsequent instances of the same task type. The distinction concerns what is compiled, how executions are consolidated, and what computation remains at test time.
To directly evaluate the benefit of this design, we added comparisons with AWM and APC, using the same source trajectories across all methods in the comparison. Entries report reward / average interaction steps.
FlatSkill achieves higher reward than all baseline methods. On ALFWorld Seen and WebShop, its reward gains over GraSP are 18.0 and 15.5 points, and its gains over AWM are 15.0 and 12.3 points. Across both ALFWorld splits and WebShop, it also uses fewer interaction steps than all three comparators. These results demonstrate the value of consolidating skill-augmented experience into a reusable task-type composition.