acceptodds
Under review as a conference paper at ICLR 2027

From Trajectories to SOPs: Reusable Procedures for Reliable Agent Execution

10 views

Abstract

Large language model (LLM)-based multi-agent systems often rely on repeated online reasoning, while the procedural knowledge in their execution trajectories remains tied to individual tasks. We propose an SOP-centric framework that abstracts shared execution logic from teacher-generated trajectories into a codebook of reusable standard operating procedures (SOPs). Each SOP specifies procedural steps, input-output contracts, and verification rules, enabling procedures to be retrieved, composed, and verified across tasks. A large-model Overseer performs SOP-level planning and dispatches compact latent codes grounded in this codebook, while lightweight Executors follow the composed procedures with runtime verification, local repair, and selective escalation. Across AgentSkillOS, SkillsBench, and SWE-bench Verified, our framework improves the macro-average score by 33.1% over the strongest baseline. With a 9B executor, it improves performance by 79.4% over Vanilla while reducing allocated GPU-seconds per task by 18.6%. Compared with natural-language plans, latent SOP dispatch reduces serialized communication payload by a factor of without sacrificing task quality. Ablation and perturbation studies support the importance of SOP abstraction, contract verification, and runtime recovery for reliable execution. These results highlight reusable procedures as an effective interface between large-model planning and lightweight execution.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Large Language Models, Multi-Agent Systems, Knowledge Reuse, Task Planning, Agent Reliability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4M4sOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

To address the limitations that the procedural knowledge in the execution trajectories is tied to individual tasks, this paper proposes an standard-operating-procedure (SOP)-centric framework that abstracts shared execution logic from teacher-generated trajectories into a codebook of reusable SOPs. A large-model Overseer performs SOP-level planning and dispatches compact latent codes grounded in this code-book, while lightweight Executors folow the composed procedures with runtime verification, local repair, and selective escalation. Empirical studies demonstrate the effectiveness of the approach across AgentSkillOS, SkillsBench, and SWE-bench Verified.

Critical strengths

S1. The paper is well-written and easy to follow.

S2. Empirical studies demonstrate the effectiveness of the proposed approach.

Critical weaknesses

The conceptual novelty of the proposed framework is somewhat unclear.

  • SOPs appear to be a specialized form of reusable agent skills, an abstraction already familiar to the community.
  • The Overseer–Executor architecture closely resembles the plan-and-execute paradigm widely adopted in modern coding agents.

The paper lacks comparisons with state-of-the-art coding-agent harnesses, such as Claude Code and OpenAI Codex.

Questions for authors

Did you also employ Qwen3.5-122B-A10B for your baselines?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 4M4s (Part 1 / 2)

For clarity in our response, we refer to the reviewer's points using the following convention: W.x (Weaknesses), Q.x (Questions), and R.x (Responses).

We sincerely thank the reviewer for the careful reading and constructive feedback. We appreciate your recognition of the paper’s clarity and the empirical evidence supporting the effectiveness of our approach. Your comments highlight the need to clarify our methodological contributions, the scope of comparisons with coding-agent harnesses, and the model configurations used for the baselines. We address the three concerns and your question below.


W.1 Novelty beyond existing skill-based methods and plan-and-execute architectures.

R.1 We agree that SOPs are closely related to reusable agent skills and can be understood within this broader family of procedural abstractions. Our methodological contribution concerns how shared execution knowledge is extracted from trajectories, organized through explicit contracts, and used to govern subsequent execution. Three aspects clarify this contribution:

  • We consolidate shared execution logic from historical trajectories into normalized and deduplicated procedures with explicit applicability conditions, input–output contracts, and verification rules. These procedures organize the use of underlying skills. Under the same historical experience, Table A.2 shows that structured SOPs improve the macro-average score from 24.17 with raw trajectory retrieval to 30.57, supporting the value of this transformation for procedural reuse.
  • The contracts are used to check whether the outputs of one procedure satisfy the inputs required by another. This structure determines how retrieved procedures can be composed into an executable dependency graph. Table A.4 isolates this organizational contribution using the same SOP set: on complex tasks, dependency-guided composition increases success from 17.0% to 26.0% and reduces invalid-input calls from 19.6% to 10.4%, compared with SOP Set Prompting.
  • During execution, verification results determine whether dependent procedures are released or local repair and escalation are triggered. The procedural representation therefore participates directly in runtime control. Table 5 supports the importance of these mechanisms: removing contract verification and runtime recovery reduces the macro-average score by 6.29 and 7.10 points, respectively. We therefore position our contribution within the broader literature on reusable skills and workflows. The contribution lies in the specific trajectory-to-SOP construction, contract-based composition, and verification-guided execution mechanisms, whose practical value is supported by the corresponding controlled comparisons and ablations.

R.2 Thank you for encouraging us to clarify our relationship to the plan-and-execute architecture. We agree that the separation between planning and execution is well established, and we adopt it as the organizational foundation of our framework. Our methodological contribution concerns the SOP-based planning, dispatch, and execution mechanisms developed within this architecture. We clarify the responsibilities and interfaces of the two roles below:

  • In a representative plan-and-execute implementation, the planner generates a task-specific sequence of steps for an executor to complete. Our Overseer performs procedural planning over a codebook constructed from historical trajectories. Its planning units are reusable SOPs with explicit applicability conditions, input–output contracts, and verification rules. The runtime grounds the selected procedures in the current task and state, then compiles them into a dependency structure according to their input–output requirements.
  • Our lightweight Executors dynamically load the selected SOPs and perform skill-level actions under their procedural policies and tool bindings. The SOP supplies reusable execution logic, while the runtime uses its contracts and verifiers to check whether execution can proceed. Verified completion releases dependent procedures; failed checks trigger local repair, alternative SOP loading, or escalation. These mechanisms connect procedural planning to explicit conditions for execution progression and recovery.
  • The Overseer communicates compact latent codes, which the runtime resolves into SOPs using a compatible codebook and the current task state. This interface supports the reuse of procedural knowledge without repeatedly transmitting its full description. In the dispatch comparison in Table 3, latent instructions reduce the serialized payload from 4,860.00 to 76.08 bytes per episode, corresponding to a reduction by a factor of 63.88, while the macro-average score increases from 24.69 to 27.10 relative to natural-language plans. These results provide empirical support for the value of this dispatch interface in the evaluated setting.
AuthorsOpenReview ↗

Response to Reviewer 4M4s (Part 2 / 2)

We therefore position our work as a contribution within the established plan-and-execute paradigm. Our contribution specifies how trajectory-derived procedural knowledge is selected, composed, communicated, and used to govern execution within this division of labor. We appreciate the reviewer’s comments, which have helped us distinguish the architectural foundation from the specific mechanisms introduced in our work.


W.2 The paper lacks comparisons with state-of-the-art coding-agent harnesses, such as Claude Code and OpenAI Codex.

R.3 Thank you for this helpful suggestion. We agree that comparisons with established coding-agent harnesses would strengthen the empirical evaluation and better contextualize our contribution. To address this point, we conducted additional evaluations of Codex CLI and Claude Code on AgentSkillOS, SkillsBench, and SWE-bench Verified. The table below reports the number of successful tasks and the corresponding success rate for each evaluated configuration.

BenchmarkToolSuccesses / TotalSuccess Rate on Evaluated Tasks
AgentSkillOSCodex CLI9/3030.00%
AgentSkillOSClaude Code15/3050.00%
SkillsBenchCodex CLI7/828.54%
SkillsBenchClaude Code12/8214.63%
SWE-bench VerifiedCodex CLI78/50015.60%
SWE-bench VerifiedClaude Code201/50040.20%
These additional experiments broaden our baseline coverage to include practical coding-agent harnesses. In the evaluated configurations, Claude Code achieves higher task-level success rates than Codex CLI across all three benchmarks. As these results are based on a single evaluation run per configuration, we have not yet included them in the manuscript. Multi-seed evaluations consistent with the protocol used in the paper are currently underway, and we will incorporate the updated results, together with the corresponding model configurations and evaluation settings, into the final version.

Q.1 Did you also employ Qwen3.5-122B-A10B for your baselines?

R.4 Thank you for the opportunity to clarify the model assignments used in our comparisons. For Plan-and-Execute, AgentSkillOS, and our AutoGen configuration, we used Qwen3.5-122B-A10B for planning or orchestration and Qwen3.5-9B for execution. These configurations explicitly separate the two roles, allowing us to match the planner and executor backbones used in our framework. For the remaining baselines, we preserved their original control flows and used Qwen3.5-9B for their LLM components, without introducing a separate 122B planner. This choice was intended to preserve the defining structure of each baseline. For example, ReAct interleaves reasoning and action generation within a single agent loop; adding a separate planner would modify that architecture, so the resulting evaluation would concern an adapted variant of the method. We recognize that these assignments and their rationale should be stated more explicitly. We will clarify them in the experimental setup to make the comparisons easier to interpret and reproduce.


References:

  1. Zora Zhiruo Wang et al. 2025. Agent Workflow Memory. Proceedings of the 42nd International Conference on Machine Learning, PMLR 267: 63897–63911.
  2. Guibin Zhang et al. 2025. G-Memory: Tracing Hierarchical Memory for Multi-Agent Systems. In Advances in Neural Information Processing Systems, volume 38.
  3. Lutfi Eren Erdogan et al. 2025. Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks. In Proceedings of the 42nd International Conference on Machine Learning, PMLR 267, pages 15419–15462.
  4. Ao Li, Yuexiang Xie, Songze Li, Fugee Tsung, Bolin Ding, and Yaliang Li. 2025. Agent-Oriented Planning in Multi-Agent Systems. In International Conference on Learning Representations.

We sincerely thank the reviewer for the thoughtful questions and constructive feedback. Your comments have helped us clarify our methodological contributions and experimental setup, improving both the clarity and overall quality of the manuscript. If these revisions address your concerns, we would be grateful if you would consider increasing your score. We would be happy to discuss any remaining questions.

Thank you for the detailed responses. I will maintain my current rating.

Reviewer_U8xMOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper distills teacher trajectories into reusable SOPs with typed inputs/outputs and verifiers. A 122B Overseer selects procedures through latent codes, and a 9B Executor runs them with verification, repair and escalation. Across AgentSkillOS, SkillsBench and SWE-bench Verified, the paper reports a 33.1% macro-average gain over the strongest baseline. The codebook contains 54 SOPs built from 1,086 trajectories.

Critical strengths
  1. The problem is well motivated and the evaluation spans three different benchmarks, multiple executor sizes and several perturbations.
  2. The appendix includes useful controls. On its held-out tasks, SOPs beat raw trajectory retrieval from the same experience (24.17→30.57 macro), and dependency-guided composition improves over prompting with the same SOP set. The paper also reports negatives, including worse results than Vanilla at 2B and 4B.
Critical weaknesses
  1. The construction/evaluation split and runtime access to grading signals are unclear. The paper never identifies Q_train for the main tables. Table 4 lists 295 AgentSkillOS-source trajectories, although the public benchmark has only 30 tasks, and 500 SWE-bench trajectories, equal to the Verified split size. This does not prove overlap, but it needs an explicit task manifest. The only clearly described held-out result is lower than the main result: 30.57 versus 34.44 macro. Separately, Table A.5 describes “native checks” and recovery from an orientation requirement that appears in the public evaluator rather than the task prompt. Are these independent SOP checks or benchmark grading signals? If evaluation tasks or hidden checks shaped construction or recovery, the transfer claim changes substantially.
  2. The comparison does not clearly match model access or compute. Only Ours is described as using the 122B Overseer for planning and escalation, despite the statement that all methods share the same models. The 122B model used as a Vanilla executor already scores 35.35, above Ours-9B's 34.44. A control with the same Overseer and escalation but no SOPs is essential; the appendix controls may provide it, but their configurations are not stated. GPU-seconds are also undefined. It is unclear whether they include the Overseer, or why Table 2 reports lower GPU-seconds while Figure 4 shows higher latency and executor tokens.
  3. The reported uncertainty and subgroup results need reconciliation. Table 1 labels “±” as SD without stating the number of runs. The AgentSkillOS and SkillsBench means fit four runs over binary tasks, but many reported SDs do not fit that interpretation, including Ours' ±1.04 and ±0.93. Table A.1 has a similar issue: pylint has only ten Verified instances, yet values such as 23.46% do not fit one or four runs. Its repository-weighted values nevertheless aggregate exactly to the main SWE-bench results. An undisclosed metric or aggregation rule might explain this; per-instance, per-run outcomes are needed before these numbers can be trusted.
Questions for authors
  1. Which task IDs produced the 1,086 trajectories? Please identify the construction and evaluation sets for every main table and explain what Table 4 coverage measures.
  2. What checks can the runtime see? Are Table A.5's “native checks” benchmark evaluators or separately written SOP verifiers, and do baselines get the same feedback?
  3. Which controls retain the same 122B Overseer and escalation without SOPs? Please define total compute accounting and reconcile Table 2 with Figure 4.
  4. How many runs were used, what does each SD measure, and how was Table A.1 computed? Please provide per-instance outcomes that reproduce both tables.
8 comments
AuthorsOpenReview ↗

Response to Reviewer U8xM (Part 1 / 5)

For clarity in our response, we refer to the reviewer's points using the following convention: W.x (Weaknesses), Q.x (Questions), and R.x (Responses).

We sincerely thank the reviewer for the careful reading and detailed feedback. We particularly appreciate your scrutiny of the reported data and statistics, which has helped us identify issues requiring correction or clarification and strengthen the transparency of our experimental reporting. We also appreciate your recognition of the motivation, breadth of evaluation, and value of the appendix controls. Below, we jointly address the three weaknesses and four questions, with particular attention to data provenance, statistical consistency, and experimental fairness.


W.1 The construction/evaluation split and runtime access to grading signals are unclear.

R.1 We thank the reviewer for highlighting the need for clearer trajectory accounting and construction–evaluation protocols. We recognize that these details were not sufficiently explicit in our original description and are important for interpreting the reported results. We appreciate the opportunity to clarify them below.

  • Clarification of Table 4 and SOP construction. As formalized in Section 3.1, an SOP is a structured workflow specification with explicit procedural steps, input–output constraints, skill/tool bindings, and verification rules. It is intended to standardize how a class of tasks is executed and help execution produce outputs that satisfy the specified requirements. Successful trajectories provide concrete examples of workflows that have met these requirements and serve as the source material for SOP construction. The trajectories reported in Table 4 were generated by Qwen3.5-122B-A10B. Our collection procedure requires at least one successful trajectory per source task; when an attempt fails, the teacher explores alternative skill combinations. Consequently, trajectory counts need not equal unique task counts. We abstract shared execution logic and associated constraints from successful trajectories, followed by normalization, clustering, deduplication, and verification. We will revise the manuscript to explain this accounting more clearly, explicitly define Coverage, and distinguish trajectory counts from unique task counts.We will also provide a task-level log of trajectory collection statistics at https://anonymous.4open.science/r/iclr2027-CEE1/.
  • Clarification of the main construction–evaluation protocol. For the main experiments, QtrainQ_{\mathrm{train}} comprises the 30 AgentSkillOS tasks, the 82 tasks in our frozen SkillsBench snapshot, and the 500 SWE-bench Verified instances. The 1,086 reported trajectories originate from these source tasks, which also appear in the main evaluation. The main experiments therefore assess procedural reuse with prior task experience. Baselines retrieve raw historical trajectories through RAG, while our framework accesses SOPs derived from that experience: the Overseer selects and composes preconstructed SOPs and assigns them to the Executor for execution.
  • Clarification of Table A.2 and its relationship to the main results. The main experiments assess procedural reuse with prior experience of the evaluated tasks, whereas Table A.2 uses a task-disjoint construction–evaluation split. In Table A.2, SOP construction and baseline retrieval use only construction-task trajectories, and the SOP codebook is frozen before evaluation. We will clarify these experimental settings in the description accompanying Table A.2 to explicitly distinguish its protocol from that used in the main experiments.
AuthorsOpenReview ↗

Response to Reviewer U8xM (Part 2 / 5)

W2 & Q3 It is unclear whether they include the Overseer, or why Table 2 reports lower GPU-seconds while Figure 4 shows higher latency and executor tokens. Which controls retain the same 122B Overseer and escalation without SOPs?

R.2 We clarify below the measurement scopes of Table 2 and Figure 4 and explain how Vanilla provides the matched control with the same Overseer and escalation mechanism but without SOPs.

  • Clarification of Table 2. We thank the reviewer for drawing our attention to the need for a clearer explanation of the compute measurements. Table 2 examines Executor scaling, and its compute-cost columns report Executor-side allocated GPU-seconds per task. Under a fixed GPU allocation, this is the elapsed wall-clock time from the Executor’s receipt of its task instructions to task termination, multiplied by the number of GPUs allocated to the Executor. This covers all elapsed time from instruction receipt to task termination, including skill loading, tool invocation, execution, verification, and waiting; the Overseer’s GPU usage is outside this accounting scope. The reported savings therefore refer specifically to Executor-side allocated GPU time. Although the caption identifies the focus on Executor scaling, it may not fully convey these measurement boundaries.
  • Clarification of Figure 4. We thank the reviewer for highlighting the need to distinguish these experiments more clearly. Figure 4 examines how different information representations affect communication and execution within a multi-agent system, with the aim of evaluating the contribution of our latent representation. Its end-to-end latency is measured from task initiation to termination, covering planning, SOP selection, and dispatch before the Executor receives its instructions, as well as subsequent execution. This measurement therefore spans a broader interval than the Executor-stage timing in Table 2. Lower Executor-side GPU time can coexist with higher end-to-end latency when additional overhead outside the Executor stage outweighs the execution-time reduction.
  • Relationship between Table 2 and Figure 4. We appreciate the reviewer’s concern about the apparent discrepancy between these results. As discussed above, Table 2 and Figure 4 report separate experiments with different objectives and measurement scopes. Table 2 evaluates Executor scaling using Executor-side allocated GPU time, while Figure 4 compares information representations within the multi-agent system and reports Executor-token consumption and end-to-end latency. The latter latency measurement also includes stages before the Executor receives its instructions. Accordingly, the Executor-side cost reduction in Table 2 does not necessarily imply corresponding reductions in latency or token consumption in the separate representation experiment.
  • Vanilla provides the matched control without SOPs. We thank the reviewer for highlighting the importance of clarifying this control. In the main experimental configuration, our Vanilla baseline retains the same Qwen3.5-122B-A10B Overseer for planning and escalation and the same Qwen3.5-9B Executor as our framework, while operating without SOP guidance. This configuration may not be sufficiently explicit in the current description. We will therefore explain Vanilla’s model assignments and escalation mechanism more clearly in the main text to make the basis of this comparison easier to assess.

AuthorsOpenReview ↗

Response to Reviewer U8xM (Part 3 / 5)

W3 & Q4 The reported uncertainty and subgroup results need reconciliation.

R.3 We sincerely thank the reviewer for carefully examining the reported statistics. Your comments prompted us to re-examine the original execution logs and the aggregation pipeline. We identified a naming collision in our logging system: filenames did not uniquely distinguish runs, allowing other processes to write records to the same files. The resulting mixture of records affected some standard deviations in Table 1 and the subgroup statistics in Table A.1. This was an oversight in our log management and validation process, and we sincerely apologize for the reporting errors and the uncertainty they have caused. We conducted 40 runs using distinct random seeds from 42 to 81, inclusive. We will explicitly state this run count in the revised experimental setup and clarify the definition of the reported SD and the subgroup aggregation procedure. We will recompute the statistics in both tables from validated per-instance, per-run outcomes and correct the affected values and related statements. We will also make a consolidated CSV log containing 367,200 per-task evaluation records available at [LINK]. The log includes benchmark and method labels, task identifiers, task categories or repositories, random seeds (42–81), binary success outcomes, and execution statuses to support independent checking of the reported aggregate and subgroup statistics. We appreciate the reviewer’s diligence in bringing this issue to our attention and helping us improve the accuracy and reproducibility of our reporting.


Q1 Which task IDs produced the 1,086 trajectories? Please identify the construction and evaluation sets for every main table and explain what Table 4 coverage measures.

R.4 We thank the reviewer for highlighting the need to explain the construction data more precisely. Our collection criterion was to obtain at least one successful teacher-generated trajectory for each target task category. For a category without a successful trajectory, the Qwen3.5-122B-A10B teacher made additional execution attempts, including attempts with alternative skill combinations. Once a category had a successful trajectory, additional task instances in that category could be omitted from further offline collection.

AspectCollection protocol
Coverage unitA target task category
Minimum requirementAt least one successful teacher-generated trajectory per category
Additional attemptsContinue attempting execution when a category has no successful trajectory
Stopping ruleFurther task instances within a category may be skipped once the minimum requirement is satisfied
SOP construction inputsSuccessful trajectories used to abstract workflows and specify input/output contracts, execution steps, and skill bindings

This category-level collection rule means that the number of generated trajectories, the number of distinct construction tasks, and the number of evaluation tasks need to be reported separately. Multiple attempts may originate from the same task, while some evaluation tasks may not have been executed during trajectory collection. We have made two types of logs available at https://anonymous.4open.science/r/iclr2027-CEE1/: statistical logs for reproducing the numerical results reported in the paper’s tables, and raw execution records for inspecting the underlying runs. The statistical logs include task-level trajectory allocations, specifying the benchmark, task identifier, and numbers of correct and incorrect trajectories allocated to each task, alongside the evaluation data used to compute the reported statistics. The raw execution records provide detailed information about individual runs to support independent inspection and verification.

  • Clarification of the construction and evaluation sets. We thank the reviewer for requesting a clearer description of the experimental protocol. Let (C) denote the successful teacher-generated trajectories retained for SOP construction, together with their source-task IDs. Let (E) denote the full evaluation set: all 30 AgentSkillOS tasks, all 82 tasks in our frozen SkillsBench snapshot, and all 500 SWE-bench Verified instances. The main performance experiments use the full evaluation set, while SOP construction draws on successful trajectories from a subset of these tasks. Trajectory collection follows the category-level criterion described above, so an evaluation task does not necessarily contribute a trajectory to SOP construction.
Reviewer_4ThWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper propose a framework to convert agentic trajectories into reusable SoPs (standard operating procedures). It contains two stages: an offline construction and an online execution. In the offline stage, a large teacher model generate trajectories, and then summarized into resuable SoPs in a codebook; in the online stage, a large model selects relevant SoPs, and a lightweight model execute the task with the selected SoPs. The framework imporves the task performance across 3 benchmarks (AgentSkillOS, SkillBench, SWE-bench).

Critical strengths
  1. The idea is useful for a practical system, as reusing the procedures from past experience reduces repeated reasoining for complex tasks.
  2. The evaluation resutls are sound across different benchmarks with multiple baselines.
  3. The ablation study further shows the contribution of each component.
Critical weaknesses
  1. The novelty of the work is limited. In essence, the work is to summarize SoPs/Skills was explored before.
  2. It would be beneficial to test if the SoPs can be transfered across different task types.
  3. Missing references: Li, Z., Huang, S., Wang, J., Zhang, N., Antoniades, A., Hua, W., ... & Yan, X. (2025). Sopbench: Evaluating language agents at following standard operating procedures and constraints. arXiv preprint arXiv:2503.08669. Chen, P., Zhang, S., & Han, B. (2024, June). Comm: Collaborative multi-agent, multi-reasoning-path prompting for complex problem solving. In Findings of the Association for Computational Linguistics: NAACL 2024 (pp. 1720-1738).
Questions for authors
  1. Does the SoP have to be a code book? How does this perform in comparison with semantic matching?
  2. Have the authors tried the training-based methods to further improve the SoP generation and execution?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 4ThW (Part 1 / 3)

For clarity in our response, we refer to the reviewer's points using the following convention: W.x (Weaknesses), Q.x (Questions), and R.x (Responses) .

We sincerely thank the reviewer for the careful reading and constructive feedback. We appreciate your recognition of the practical value of reusing procedural knowledge, the breadth of our evaluation, and the contribution of the ablation studies. Your comments also highlight aspects of the contribution and experimental scope that deserve a clearer explanation. We address the three weaknesses and two questions below.


W.1 & W.3. Novelty and related work.

R.1 We thank the reviewer for encouraging us to clarify our contribution and its relationship to prior work on experience reuse. ExpeL (Zhao et al., 2024) extracts textual insights and retrieves successful trajectories. AWM (Wang et al., 2025) induces recurring workflows for memory augmentation and explores callable workflow actions. LEGOMem (Han et al., 2025) distributes task-level and subtask-level memories to support orchestration and tool use. These approaches provide important foundations for our work. We also appreciate the reviewer’s recommendations of SOPBench (Li et al., 2025) and CoMM (Chen et al., 2024), both of which we have incorporated into Related Work. SOPBench evaluates compliance with specified procedures using rule-based oracles, while CoMM organizes collaborative reasoning through complementary roles and reasoning paths. Building on these foundations, our contribution centers on three closely connected aspects of converting lengthy trajectories into reusable procedural knowledge:

  1. Trajectory-to-SOP abstraction. ExpeL extracts textual lessons, while AWM induces recurring routines. Our work consolidates shared execution logic into normalized, deduplicated SOPs with explicit applicability conditions, input–output contracts, and verification rules. This representation specifies both when a procedure is applicable and how its successful completion can be verified. With the same historical experience available to both approaches, Table A.2 reports a macro-average increase from 24.17 with raw trajectory retrieval to 30.57 with structured SOPs. This comparison provides evidence that the proposed representation helps make more effective use of the available trajectories.
  2. Retrieval and composition. Voyager retrieves and composes executable skills (Wang et al., 2024). Our work builds on the broader idea of reusable skills by organizing retrieved SOPs through checked input–output dependencies, helping prevent calls whose input requirements have not been satisfied. With the same SOP set, Table A.4 shows that dependency-guided composition improves success on complex tasks from 17.0% to 26.0% and reduces invalid-input calls from 19.6% to 10.4%, compared with SOP Set Prompting. These results support the additional value of explicitly organizing dependencies among the available procedures.
  3. Execution in multi-agent systems. LEGOMem retrieves memories to assist an orchestrator in generating subsequent subtasks. In our framework, the Overseer composes SOPs for lightweight Executors, while the runtime uses verification results to release dependent procedures and trigger local repair or escalation. This connects reused procedural knowledge to explicit conditions for execution progression and recovery. Table 5 reports macro-average drops of 6.29 and 7.10 points when contract verification and runtime recovery are removed, respectively, providing empirical support for the role of these mechanisms. Our methodological contribution therefore lies in how shared trajectory knowledge is represented, composed, and executed under explicit contracts, with the above comparisons supporting the practical value of these mechanisms. We appreciate the reviewer’s comments, which have helped us articulate these distinctions more clearly and better position our contribution within the existing literature.

AuthorsOpenReview ↗

Response to Reviewer 4ThW (Part 2 / 3)

W.2 Transfer across different task types.

R.2 To better understand the extent to which SOPs can be reused across task types, we evaluated four settings on AgentSkillOS: execution without SOPs, with same-type SOPs, with cross-type SOPs, and with a mixture of both. Our results suggest that same-type SOP reuse provides substantial benefits, while the gains from cross-type reuse are more limited in the evaluated settings.

TypeNo SOPSame-type SOPCross-type SOPMixed-type SOP
Data Computation33.3360.0036.6763.33
Document Creation33.3356.6736.6756.67
Motion & Video23.3346.6720.0050.00
Visual Creation26.6753.3330.0053.33
Web Interaction36.6760.0036.6760.00
Macro Avg.30.6755.3332.0056.67

These results help us better characterize both the benefits and the scope of our approach:

  1. Effectiveness of same-type reuse. Same-type SOPs improve the macro-average score from 30.67 to 55.33, with improvements across all five task types. These findings support our central objective of reusing shared execution logic from historical trajectories by organizing skills into procedures with explicit dependencies, input–output contracts, and verification rules.
  2. Limited benefits of cross-type reuse. Cross-type SOPs achieve a macro-average score of 32.00, compared with 30.67 without SOPs. The improvements are modest and vary across task types: Web Interaction remains unchanged, while Motion & Video declines from 23.33 to 20.00. Mixed-type SOPs reach 56.67, a small numerical increase over 55.33 with same-type SOPs. We therefore interpret these findings cautiously: they support procedural reuse when tasks share relevant execution logic, but do not establish broad transfer across different task types.
  3. Possible role of underlying skills. One possible explanation for the limited cross-type improvements is that some underlying skills remain useful across tasks, even when the execution order or dependencies encoded in their original SOPs are less applicable. However, the present comparison does not independently isolate skill-level and SOP-level contributions. We therefore offer this as a tentative interpretation, rather than a conclusion established by the experiment. These findings support our focus on organizing skills and trajectory-derived execution logic for reliable procedural reuse, while also helping us identify its limitations. In particular, the usefulness of individual skills across tasks does not necessarily imply that their SOP-level organization transfers equally well. We have incorporated these findings into the Limitations section to explicitly acknowledge the limited benefits of cross-type SOP transfer observed in our evaluation. We appreciate the reviewer’s comments, which have helped us clarify the scope of our contribution and communicate this limitation more clearly.

AuthorsOpenReview ↗

Response to Reviewer 4ThW (Part 3 / 3)

Q.1 Codebook design and comparison with semantic matching.

R.3 Thank you for the opportunity to clarify the role of latent coding in our framework. We agree that an SOP does not inherently require a discrete latent code, and a codebook is compatible with semantic matching. To make this distinction clearer, we separate the structured SOP repository from the mechanisms used to retrieve and dispatch its entries:

  • In Section 3.1, the codebook refers to a collection of structured SOP entries containing procedural policies, input–output contracts, tool bindings, and verifiers. Semantic matching can retrieve procedures from this collection. Our discrete latent representation provides a compact interface for dispatch and grounding, and the current implementation requires a compatible codebook to resolve the transmitted codes.
  • Section 4.4 and Figure 5 compare FSQ+Encoder with the trained-encoder baseline. FSQ+Encoder achieves R@1, R@5, Contract Pass, and Verified Accuracy values that are 1.25×, 1.02×, 1.14×, and 1.15× those of the baseline, respectively. These results suggest that, in this comparison, the principal benefit is more precise and execution-compatible SOP selection, while the improvement in top-k recall is modest. As described in Section 3.3, grounding considers code distance alongside input–output compatibility and tool/executor availability.
  • Table 5 shows that removing the latent code yields a macro-average score of 31.91, compared with 34.44 for the full framework. This indicates that SOP-based execution remains possible without latent coding, while providing evidence of an additional benefit from the evaluated latent-code design. We therefore view FSQ as a complement to semantic retrieval within the SOP framework. The structured codebook supplies reusable procedural knowledge, while latent coding and grounding support compact dispatch and execution-aware selection. We appreciate the reviewer’s comments, which have helped us clarify the distinct roles of these components.

Q.2 Training-based improvement of SOP generation and execution.

R.4 Thank you for suggesting training-based optimization of SOP generation and execution. We agree that additional training could improve both stages. Our current study does not apply additional task-specific training to the generation or execution models, although it does use a trained encoder for SOP grounding.

Our primary objective is to investigate how historical trajectories can be converted into explicit procedural knowledge to improve subsequent execution. Evaluating generation and execution without additional task-specific adaptation allows us to examine the contribution of SOP abstraction, organization, and runtime guidance under matched model configurations. Incorporating additional training would be a valuable extension, with controlled comparisons needed to distinguish the contributions of procedural reuse and model adaptation.

We see several promising directions. Supervised fine-tuning on validated trajectory–SOP pairs could improve the consistency of procedural abstraction, dependency specification, and contract generation. Training Executors on SOP-conditioned execution traces could also improve procedural adherence and tool-use decisions, while verifier feedback could provide useful signals for refining execution and recovery. These approaches could complement the benefits of explicit SOP organization and runtime control.

In future work, we plan to investigate training-based optimization of both stages, including supervised fine-tuning for SOP generation and Executor training using validated execution traces and verifier feedback. We will assess their incremental benefits over the current framework alongside the associated training costs. We appreciate the reviewer’s suggestion, which highlights a valuable direction for extending this work.


References:

  1. Andrew Zhao et al. 2024. ExpeL: LLM Agents Are Experiential Learners. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17): 19632–19642.
  2. Zora Zhiruo Wang et al. 2025. Agent Workflow Memory. Proceedings of the 42nd International Conference on Machine Learning, PMLR 267: 63897–63911.
  3. Dongge Han et al. 2025. LEGOMem: Modular Procedural Memory for Multi-agent LLM Systems for Workflow Automation. arXiv preprint arXiv:2510.04851.
  4. Zekun Li et al. 2026. SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints. Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing, 21561–21616.

We sincerely thank the reviewers for their time and constructive feedback, which have helped us clarify our contributions and better communicate the scope and limitations of our work. If these revisions address your concerns, we would be grateful if you would consider increasing your score. We would be happy to discuss any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.