acceptodds
Under review as a conference paper at ICLR 2027

SKT: Self-Evolving Agent Skill Library with Plan-Driven Test-time Self-Correction

Abstract

Long-horizon LLM agents must plan and act over many environment turns, yet adapting them by updating parameters is costly and often impossible behind an API. Inference-time learning from the agent's own trials is the standard alternative, but memory-based methods keep experience as episodic text that is never compiled into procedures, code-based skill libraries apply only where actions are code, and step-level verifiers deliver their verdicts through the prompt, a channel that a small frozen policy can attenuate or ignore. To address this, we propose SKT, which evolves a natural-language procedural skill library from the agent's own successes and failures, consolidates failures across trials into new skills, and seats the same frozen weights in a separate verifier context whose corrections are executed directly as environment actions. Under a matched evaluation protocol with a frozen Qwen2.5-7B-Instruct, SKT consistently outperforms inference-time baselines on held-out ALFWorld tasks, approaches an RL-fine-tuned reference without any weight update, and ports unchanged to multi-hop QA and, on a commercial API backbone, to WebShop.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Self-improvment, Long-horizon Decision Making, Test-time Self-Correction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FiHzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies inference-time adaptation of frozen language-model agents through an evolving procedural skill library and plan-driven correction. The Skill Engine creates skills from successful trajectories, revises retrieved skills after failures, and periodically consolidates failure lessons across trials. The Critique Engine uses the same frozen backbone in a separate context to verify plan-step completion and directly execute corrective actions. On ALFWorld, SKT achieves 58.96±1.50% success with Qwen2.5-7B-Instruct under the authors’ matched protocol, compared with 55.72±1.14% without the Critique Engine. The paper also evaluates the framework on multi-hop QA, WebShop, and additional backbones, and includes operator ablations and an advice-versus-execution comparison.

Critical strengths

1.The main empirical comparison controls important sources of variation. The inference-time baselines share the backbone, maximum number of attempts, and action budget, with experience sources documented separately (§4.1; Appendix B.2). Both full SKT and the library-only configuration outperform these baselines across all three reported evaluation seeds (Tables 2–3). This makes the observed improvement more persuasive than a comparison assembled from different repositories or evaluation protocols. 2.The paper makes a meaningful effort to identify which mechanisms contribute. Rebuilding the library without each operator establishes their practical contribution (Table 7). More distinctively, Table 15 compares neutral text, corrective advice, and direct execution using the same verifier design; direct execution exceeds advice on all three seeds, by 2.24 percentage points on average. This is useful evidence for the proposed correction interface and is the paper’s clearest empirical contribution.

Critical weaknesses

1.The distinctive contribution of the Skill Engine is not sufficiently established against closely related procedural-memory methods. The framing in the abstract and §2 understates prior work on structured experience and failure-driven revision. AutoManual creates, updates, and consolidates success/error rules; ReasoningBank extracts structured strategies from successful and failed experiences; and Memᵖ revises retrieved procedural memories using failed executions. The more specific contribution appears to be support-filtered, cross-task failure consolidation combined with plan-triggered direct correction. However, the current comparisons do not establish which benefits arise from this particular consolidation mechanism, rather than more generally from learning and revising procedural knowledge. This matters because it determines the extent of the methodological contribution beyond the demonstrated system improvement. Relevant sources: AutoManual, §3.3–3.4, ReasoningBank, §3.2–3.3, and Memᵖ, §4.3. 2.The central correction controller cannot be unambiguously reconstructed from the supplied specification. Appendix E permits next_action only when fix_worked=false. Yet the worked example requires a successful prerequisite correction—opening the microwave—to be followed by another corrective action before the original heating step is complete (pp.20–21, 29–30). Algorithm 1 also exits the loop on replanning, whereas Table 14 describes generating a fresh plan within the current attempt. These are inconsistencies in the written specification; they do not establish that the executed code is incorrect. Nevertheless, the alternatives produce different action sequences and budget handling, so resolving them is necessary to assess and reproduce the principal correction mechanism.

Questions for authors

1.Contribution clarification: What precisely distinguishes the failure-driven library updates from AutoManual, ReasoningBank, and Memᵖ? For the Distiller, is there existing evidence or a targeted control using the same failure lessons through sequential refinement, separating cross-trial consolidation from additional calls or changes in library size? 2.Implementation clarification: How does the actual controller handle fix_worked=true with now_landed=false? Does replanning continue the current attempt, and how are environment state, history, and remaining action budget preserved? Please reconcile the implementation with Algorithm 1 and the printed prompts. 3.Channel-control clarification: In Table 15, does “same firing schedule/context” mean the same triggering rule or matched realized events? Since direct execution changes subsequent observations, please clarify tracker advancement and verify-after-fix handling across arms, preferably with existing call-count statistics. 4.Robustness clarification: Appendix B.1 states that Tables 2 and 15 reuse one frozen library per configuration, while Tables 6 and 7 rebuild libraries per seed. Which results share library artifacts or evaluation runs, and what variability is observed across independently constructed libraries? This would clarify whether the reported robustness extends to the skill-evolution process itself.

7 comments

We thank Reviewer FiHz for the careful and precise review. We answer Questions 1–4 in 5 comments; W1 is answered with Q1 and W2 with Q2.

Q1 / W1 (1/2) — What distinguishes the failure-driven library updates from AutoManual, ReasoningBank and Mem$^p$

The difference is in the path a failure takes into the library, at two points: the Refiner decides between patching the retrieved skill and skipping, and the Distiller consolidates failures from different trials under a support threshold. The table sets this against the sections the reviewer cites ("—": not described in that paper).

MethodWhat a failed episode changesFailures aggregated across tasksMinimum support
AutoManual (§3.3)the Builder writes or updates rules from that trajectory; a failed task yields an "Unsolved Error" ruleno; the Consolidator merges or deletes rules when their number exceeds a cap (12 in its experiments)—
ReasoningBank (§3.2)a trajectory judged as failed is distilled into memory items (pitfalls, counterfactual signals)no; new items are "directly added without additional pruning" (App. A.2); MaTTS (§3.3) contrasts several trajectories of one query—
Mem$^p$ (§4.3)Adjustment: the failed trajectory "is combined with the original memory and then revised in place"no; the periodic Validation step keeps successful trajectories only—
SKTRefiner: appends an anti-pattern (optionally an instruction step) to the most relevant retrieved skill, or skips when the failure is attributable to the base model rather than to the skill; the lesson joins a bufferyes; every Nfire=20N_{\mathrm{fire}}=20 buffered lessons the Distiller clusters them by dominant failure axis and, for each cluster, refines an overlapping skill or creates a new oneclusters below Ncluster=3N_{\mathrm{cluster}}=3 lessons are dropped as noise
  • Refiner. As in Mem$^p$'s Adjustment, the entry that was used is revised. The Refiner adds the patch-or-skip decision, which keeps failures of the base model from being written into a skill, and it passes every lesson to the buffer that the Distiller reads.
  • Distiller. AutoManual's Consolidator is triggered by the size of the manual and merges existing rules. The Distiller is triggered by accumulated failures and changes the library only when at least three lessons share a failure axis; this yields skills such as early-termination and lost-track-of-progress, which no success-triggered extractor can write. In the three papers we find no step that aggregates failures across tasks under a support threshold.

Revised framing. The abstract, Section 2 and Table 1 will describe prior work on structured experience and failure-driven revision as follows.

  • Abstract: "memory-based methods keep experience as episodic text that is never compiled into procedures" becomes "memory-based methods mostly store experience as reflections, insights or context-resident rules".
  • Section 2 will state the contrast in how failure enters the store: ExpeL turns a failure into an insight, ReasoningBank into a memory item appended to the bank, Mem$^p$ into a revision of the entry that was used; SKT's refiner patches the specific procedure that misled the agent, and its distiller clusters many failures, with a support threshold, into a new procedure.
  • Table 1 will add the closest methods, and its caption no longer claims uniqueness:
MethodExperienceFrom failureConsolidates failuresVerificationBackbone
AutoManualrule manualerror rulesrules merged above a cap—GPT-4-turbo
ReasoningBankreasoning memory itemsjudged failures as pitfallsfalse (append only)—Gemini-2.5, Claude-3.7
Mem$^p$procedural memoryrevised in place after a failed usefalse—GPT-4o, Claude, Qwen2.5-72B
SKT (ours)evolved NL procedurespatch + distilltruedirect actuationQwen2.5-7B (frozen)

Revised caption: "SKT combines natural-language procedural skills evolved from both success and failure, consolidation of failures across trials, and verification by direct actuation, with a frozen open-weight backbone; to our knowledge, none of the listed methods combines all three."

Q1 / W1 (2/2) — Evidence for the Distiller: existing evidence and a targeted control

Existing evidence. In the leave-one-out ablation (Table 7) the library is rebuilt without the Distiller while the Refiner stays, so every failure is still processed sequentially, one trial at a time. Success drops from 58.96±1.5058.96\pm1.50 to 53.23±3.5353.23\pm3.53 (−5.73±2.15-5.73\pm2.15 pp, 3/3 seeds), and the best ablated seed is below the worst seed of the full system. The three Distiller-origin skills (incorrect-task-decomposition, early-termination, lost-track-of-progress) are the only cross-family entries of the 19-skill library. This ablation compares sequential refinement alone with sequential refinement plus consolidation; it does not separate consolidation from the additional Distiller calls or from the size of the library, which is what the question asks.

Targeted control (new). We ran the variant described in the question, in which failure lessons are processed one at a time: every failure lesson of attempts 2 and 3 triggers its own Distiller call (fire threshold 1, minimum cluster size 1) with the library snapshot in context, instead of being buffered until 20 lessons accumulate and then clustered by failure axis (minimum cluster size 3). We compare with SKT as reported in Table 2.

ConfigurationDistiller callsskillsseed 41seed 42seed 43mean ± std
original batch Distiller91957.4658.9660.4558.96 ± 1.50
per-lesson Distiller1392452.2461.9455.9756.72 ± 4.89
  • Success. The control reaches 56.72±4.8956.72\pm4.89 against 58.96±1.5058.96\pm1.50 for SKT; the difference is −5.22-5.22, +2.98+2.98 and −4.48-4.48 pp for seeds 41, 42 and 43, so per-lesson processing does not improve on consolidation.
  • Calls. The control makes one Distiller call per lesson, where the batch Distiller makes one call per 20 lessons.
  • Library size. The control ends with 24 skills, against 19 in SKT's library.

More calls and a larger library therefore do not raise success above SKT's. We present consolidation as the low-cost form of turning failure lessons into skills. The revision will report this control in Appendix D.3.

Q2 / W2 — The correction controller

We thank the reviewer for reading the controller specification this closely. We answer the points of W2 and Q2 one by one, in the order of the review, and note where the paper already gives the rule.

1. fix_worked = true with now_landed = false (Q2). This is the case in which the corrective action took effect but the plan step itself is not yet achieved, for example when the corrective only opens a receptacle that the step needs. In this case the pointer stays at the current step, no further corrective action is issued, and the policy takes the next action from the corrected state. The paper already states both parts: the fix-verify prompt (Appendix E) gives next_action "Only when fix_worked=false AND should_escalate_replan=false. Else null", and in Algorithm 1 only now_landed advances the pointer (line 10); otherwise line 6 takes the next action from the policy.

2. Worked example and the prompt (W2). The reviewer notes that the prompt permits next_action only when fix_worked is false, while the worked example reads as if a second corrective action followed a successful one. The prompt is what applies here: after open microwave 1 succeeds, the subsequent heat tomato with microwave 1 is emitted by the policy, not by the verifier. The worked example writes the two actions as one sequence, which is why it reads as a second corrective; the revision will replace it by the annotated trace below.

3. Does replanning continue the current attempt? (W2, Q2) The reviewer also notes that Algorithm 1 leaves the loop at "flag replan; break", while Table 14 describes a fresh plan within the current attempt. Replanning does continue the current attempt. Table 14 (the events of the Critique Engine) in Appendix D.2 already gives the escalation ("Escalate": re-issue the first-turn plan directive, c←1c \leftarrow 1); Algorithm 1 does not show it, so we clarify it here.

4. Environment state, history and action budget on replanning (Q2). Because replanning stays within the attempt, nothing is restarted: the environment is not reset, the action–observation history is kept, the action counter continues, and every action, corrective or not, is charged to the budget TT.

Complete rule. So that the controller can be reconstructed from the text alone, the table below lists every outcome of the verify-after-fix call that follows an executed corrective; the controller applies the first matching row. The revision will add this table to Appendix D.2 and an "otherwise" case to Algorithm 1:

RowVerifier outputController action
1now_landed truec←c+1c \leftarrow c+1; control returns to the policy
2now_landed false, next_action givenqueue it; it is executed as the next environment action and followed by another verify-after-fix call
3now_landed false, no next_action, should_escalate_replan trueraise the replan flag, c←1c \leftarrow 1; the first-turn plan directive is re-issued at the next policy prompt
4otherwise (e.g. fix_worked true, now_landed false)cc unchanged; control returns to the policy

Annotated example. The case of point 2, written out event by event (illustrative, written by hand to follow the prompts of Appendix E; plan step 3 is heat tomato in microwave 1):

EventVerifier outputController
The policy emits heat tomato with microwave 1; observation "Nothing happens."step check: landed=false, corrective_action="open microwave 1"execute open microwave 1 as the next environment action
Observation: the microwave is now openfix-verify: fix_worked=true, now_landed=false, next_action=null, should_escalate_replan=falserow 4: cc stays 3; control returns to the policy
The policy emits heat tomato with microwave 1; observation: the tomato is heatedstep check: landed=truec←4c \leftarrow 4
Reviewer_zNoXOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SKT, a framework that improves frozen LLM agents through a self-evolving skill library and test-time self-correction. skill engine learns and updates procedural skills from successful and failed trajectories, while the critique engine checks the agent's plan and directly executes corrective actions. Experiments on ALFWorld, multi-hop QA, and WebShop show improvements over several inference-time baselines.

Critical strengths
  1. Good experimental results.The paper compares with several baselines under a matched evaluation setup. The improvements are fairly consistent, especially on ALFWorld.

  2. Useful ablation studies. The authors analyze the different skill Engine operators and the Critique Engine. The comparison between prompt-based advice and direct execution is particularly interesting.

Critical weaknesses
  1. Limited novelty. The skill engine combines existing ideas such as skill extraction, refinement, and failure distillation, which are closely related to AutoManual, ACE. The direct execution of corrections is interesting, but I feel the overall algorithmic contribution is still limited.

  2. Limited evidence for continual self-evolution.The skill library is built on a fixed task pool and frozen during evaluation. This does not really show long-term continual improvement. I am very interested at the OOD evalution for the evolving.

  3. Generality of the Critique Engine is unclear. Most detailed analysis of the Critique Engine is limited to ALFWorld. from my experience, the behaviour of QA and webshop is very different from Alfworld and the difficulty of 2 datasets couldn't be put on the same level with alfwolrd. Also, only around 25–33% of corrective actions achieve their targets, which raises some concerns about its reliability.

Questions for authors
  1. What is the key novelty compared with existing skill learning methods like AutoManual? Will u consider to combine training method with your co-evolving TTS method?
  2. Could you test longer-term skill evolution with continuously arriving tasks and independently OOD libraries?
  3. Could you provide Critique Engine ablations on QA and WebShop, and further discuss the impact of incorrect corrections?
6 comments

We thank Reviewer zNoX for the thoughtful review. We answer the three weaknesses and questions in turn.

W1 / Q1 — Novelty relative to AutoManual and ACE; combination with training

SKT differs from AutoManual and ACE along three axes.

  1. What is stored. Ordered natural-language procedures (SKILL.md files with instructions and anti-patterns), retrieved per task; AutoManual writes rule manuals, and ACE grows an in-context playbook that accumulates in the prompt. How each method stores its content and forms the context is compared below the list.
  2. How failure enters the store. The Refiner patches the specific retrieved skill that misled the agent (and skips when the failure is attributable to the base model rather than the skill). The Distiller clusters buffered failure lessons across trials by dominant failure axis, drops clusters below Ncluster=3N_{\mathrm{cluster}}=3 members as noise, and refines or creates skills; this yields skills such as early-termination and lost-track-of-progress, which no success-triggered extractor can write. Both operators have controlled evidence: rebuilding the library without the Creator, Distiller or Refiner lowers success on all three seeds (−10.95-10.95, −5.73-5.73, −3.98-3.98 pp; the best seed of each ablated configuration is below the worst seed of the full system), and the three Distiller-origin skills (incorrect-task-decomposition, early-termination, lost-track-of-progress, retrieved in 13, 8 and 3 of the 140 collection trials) are the only cross-family entries of the 19-skill library.
  3. How verification is delivered. A separate verifier context whose verdict is executed rather than injected into the prompt; in the same-signal control, executing the verdict adds +2.24+2.24 pp (3/3 seeds), while neutral text and advice add +0.25+0.25 and +0.75+0.75 pp, neither sign-consistent.

What each method stores and how it forms the context (from the three papers and their prompts):

UnitA unit consists ofHow the context is formed
SKTworkflow (SKILL.md)a name, a one-line description, numbered steps (mostly for one task family, in class nouns such as object, receptacle) and anti-patternsup to k=3k=3 skills retrieved per task and spliced into the policy's system prompt, however large the library grows
AutoManualruleone of six types, from environment phenomena and mechanisms to success processes, helper methods and corrected or unsolved errors, a content text, and an example or code from the building trajectoriesthe Planner reads the whole manual (at most 12 rules in the paper's experiments), grouped by scenario, plus the stored code of the most similar task
ACEbulletan id, helpful/harmful counters and a short unit (a strategy, a concept, a failure mode or a code snippet) filed under a named sectionthe playbook is inserted into the Generator prompt; bullets are appended and de-duplicated, so it accumulates up to a pruning threshold

In short, SKT's context is chosen per task and each of its units is an ordered procedure. AutoManual's is one document that mixes facts, procedures and code, illustrated by examples from the building trajectories; ACE's is a sectioned list of separately editable items with helpful/harmful counters.

All roles are served by the same frozen backbone with no stronger teacher model, whereas AutoManual relies on a GPT-4-class model. The revision will state this comparison in Table 1 and Section 2. New rows of Table 1 :

MethodExperienceFrom failureConsolidates failuresVerificationBackbone
AutoManualrule manualerror rulesrules merged above a cap—GPT-4-turbo
ReasoningBankreasoning memory itemsjudged failures as pitfallsfalse (append only)—Gemini-2.5, Claude-3.7
Mem$^p$procedural memoryrevised in place after a failed usefalse—GPT-4o, Claude, Qwen2.5-72B
SKT (ours)evolved procedurespatch + distilltruedirect actuationQwen2.5-7B (frozen)

Caption: SKT combines natural-language procedural skills evolved from both success and failure, consolidation of failures across trials, and verification by direct actuation, with a frozen open-weight backbone; to our knowledge, none of the listed methods combines all three. Entries for AutoManual, ReasoningBank and Mem$^p$ follow their papers, and "—" marks no such component.

Combination with training (Q1). SKT is orthogonal to parameter updates: θ\theta is fixed by design to model the closed-API regime, and the skill library and the corrective execution live entirely at inference time. The conclusion of the revision adds: "combining it with training-time methods, for example by co-evolving the skill bank with the policy as in SkillRL, is a natural direction."

W2 / Q2 — Continual evolution and OOD (1/2)

Learning curve. Learning curve. To evaluate how held-out performance changes with the amount of experience used to construct the skill library, we rebuilt the ALFWorld library from nested prefixes of the 140 collection tasks (0, 35, 105, and 140 tasks, in type-balanced order). All results are reported as means ± sample standard deviations over three random seeds on the 134 held-out tasks.

Collection tasks035105140
success %49.75 ± 6.2153.48 ± 1.9754.98 ± 2.6258.96 ± 1.50

(i) Evolved libraries improve held-out performance. Relative to the empty-library baseline with critique engine enabled (49.75 ± 6.21), success increases by +3.73 pp with 35 collection tasks, +5.23 pp with 105 tasks, and +9.21 pp with 140 tasks. These results suggest that even a relatively small amount of collected experience can yield useful transferable skills.

(ii) Gains are concentrated in tasks requiring multi-step procedures. On heat and clean tasks, the evolved libraries solve an average of 11.5 additional tasks compared with the empty-library baseline. This finding suggest that the evolved skills are particularly beneficial for tasks involving structured, multi-step interactions.

(iii) Performance improves with additional collection experience, although seed variability remains. Increasing the collection set from 35 to 140 tasks raises mean success from 53.48% to 58.96% (+5.48 pp), with an intermediate result of 54.98% at 105 tasks. We therefore interpret these results as evidence that skill evolution provides benefits even with limited collection experience, with further gains observed as the library grows, rather than as conclusive evidence of continuous or monotonic scaling. We will incorporate this analysis into Appendix D.3 in revision.

W2 / Q2 — Continual evolution and OOD (2/2)

Continuously arriving tasks. Starting from the frozen 140-task library, we evaluate continual skill evolution as the 134 held-out tasks arrive sequentially. The Skill Engine continuously updates the library by incorporating newly acquired skills and updating while new attempts, while the Critique Engine remains enabled, with up to three attempts per task.

Configurationseed41seed42seed43mean±std
success %59.7061.9464.1861.94 ± 2.24

Continual skill evolution achieves a success rate of 61.94 ± 2.24%. Compared with the frozen-library baseline (58.96% respectively), streaming evolution yields improvements of +2.98 pp, suggests that updating the skill library between attempts may provide additional opportunities to leverage newly acquired experience. These results indicate that continual skill evolution can improve performance during deployment, particularly when skills are updated and reused across successive attempts. We will incorporate these findings into Appendix D.3.

W3 / Q3 — Critique Engine outside ALFWorld (1/2) The tracker is defined over action types rather than ALFWorld-specific vocabulary. These types correspond to environment verbs in ALFWorld, search and answer in multi-hop QA, and search, click, and buy in WebShop. Thus, adapting the tracker to a new environment requires only updating the action-type mapping. To evaluate the generalizability of the Critique Engine beyond ALFWorld, we conducted ablations across four multi-hop QA benchmarks.

BenchmarkSKT w/o Critique EngineSKT
HotpotQA27.63 ± 1.0032.02 ± 0.50
2WikiMultiHopQA14.18 ± 2.6318.83 ± 1.23
MuSiQue10.61 ± 0.5614.93 ± 0.65
Bamboogle27.74 ± 7.4333.94 ± 7.16

The Critique Engine consistently improves mean exact-match performance across all four benchmarks when combined with the Skill Engine, yielding gains of +4.39 pp on HotpotQA, +4.65 pp on 2WikiMultiHopQA, +4.32 pp on MuSiQue, and +6.20 pp on Bamboogle. These improvements suggest that the Critique Engine provides benefits beyond environment-specific action tracking and can complement skill retrieval across different multi-hop reasoning tasks. Notably, the gains on HotpotQA, 2WikiMultiHopQA, and MuSiQue are accompanied by relatively low seed-level variability in the full SKT configuration. We therefore interpret the results as evidence of improvements in mean performance across benchmarks.

WebShop. Due to budget constraints, we conducted the Critique Engine ablation using DeepSeek-V4-Flash via OpenRouter with DeepInfra as the inference provider (FP8 precision), rather than DeepSeek's official API used in Table 5. This difference in inference providers and numerical precision may account for the slight variation in success rates compared with the results reported in the main paper.

ConfigurationSeed 41Seed 42Seed 43Mean ± Std
SKT w/o Critique Engine17.218.818.618.20 ± 0.87
SKT20.620.220.220.33 ± 0.23

The full SKT framework achieves 20.33 ± 0.23% success, outperforming SKT without the Critique Engine (18.20 ± 0.87%) by +2.13 pp, with consistent improvements across all three seeds (+3.4, +1.4, and +1.6 pp). These results demonstrate that both engines contribute to WebShop performance, with the Critique Engine providing additional gains beyond skill retrieval and reuse.

W3 / Q3 — The 25–33% figure and the impact of incorrect corrections (2/2)

The 25–33% figure measures, for every corrective action of the three evaluation seeds, whether the targeted condition of the step held afterwards: 32.7% of 898 correction records under the lenient rule, and 24.75% of 711 corrections under the strict rule that counts a correction as ineffective whenever its repeated examinations disagree. The six classes of the 898 records:

ClassCountShare
effective29432.7%
re-navigation25828.7%
non-admissible command (rejected by the environment)23526.2%
precondition unmet879.7%
misdirected (wrong object or place)192.1%
indeterminate50.6%

Of the 604 non-effective records, only 19 records (2.1%) act on a wrong object or place. The net effect after paying for every correction is the paired gain of the Critique Engine, positive in all nine backbone × seed cells, at about 3% wall-clock overhead. We will include this discussion in the revision.

Reviewer_qTDZOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes SKT, a parameter-free framework that improves agent performance through a self-evolving natural language skill library and plan-driven online error correction. The skill library accumulates, revises, and consolidates experience from both successful and failed trajectories, while the correction module leverages an isolated context of the same backbone model to verify plan steps and directly execute corrective actions. The authors evaluate this framework across tasks including ALFWorld, multi-hop QA, and WebShop.

Critical strengths

The paper reports experimental results across multiple random seeds, presenting per-seed performance and paired differences rather than relying solely on best single-run results. This strengthens the credibility of the empirical findings and facilitates assessing the stability of both the overall gains and individual component contributions.

Critical weaknesses

Weakness 1

The Verifier directly executes corrective actions and subsequently appends the action and environment feedback to the interaction history, without providing the base policy with the rationale for intervention or explicit plan synchronization signals. Although the policy observes the updated state, this does not guarantee that it can properly reconcile the new state with its original plan. Local corrections may disrupt the execution sequence or conflict with long-term goals, leading to redundant actions, reversal of corrections, or inconsistent subsequent decisions. The paper neither sufficiently analyzes post-handoff behaviors nor clarifies when the agent should resume the original plan versus trigger replanning. I would like to see quantitative analyses of post-intervention plan consistency, action redundancy, and recovery success rates, alongside ablation studies that explicitly synchronize intervention rationales or update the plan upon intervention.

Weakness 2

The rule-based Tracker triggers verification based on the action type and target specified in the current plan step. If the generated plan contains omissions, misordered steps, or infeasible subgoals, the Tracker may fail to trigger or advance, or repeatedly trigger corrections around a flawed step. Because the Verifier also conditions its inspection on the same plan, it may persistently attempt to fulfill an ill-posed local subgoal without detecting structural flaws in the overall plan. Furthermore, the Verifier shares identical model weights with the policy; while an isolated context may improve local judgment, it does not guarantee identifying erroneous assumptions shared by both modules. Although the framework incorporates post-correction verification, replanning, and an action budget cap, these mechanisms primarily serve as fallback and termination safeguards rather than demonstrating an ability to promptly escape faulty plans. Further analysis is needed on recovery capability under flawed initial plans, as well as the frequency of repetitive corrections and replanning loops, rather than reporting only average gains across standard runs.

Weakness 3

In Table 2, introducing the skill library alone improves the success rate from 45.02% to 55.72%, whereas the full framework further raises it to 58.96%, indicating that the skill library accounts for the majority of the improvement. Given the extensive existing literature on experience memory and skill libraries, the paper needs to more clearly establish the unique contribution of its skill construction and updating mechanisms. Meanwhile, the core motivation for the Verifier—preventing the policy from ignoring critique feedback by directly taking over execution—addresses a general agent execution issue rather than a problem specific to skill libraries. Its inputs consist primarily of the plan and recent interactions, without explicitly verifying the preconditions or execution rules of retrieved skills. Consequently, it remains unclear whether the two modules form an essential synergy or merely function as a combination of two orthogonal, general-purpose mechanisms. I recommend adding a "Verifier-only (without skill library)" ablation and integrating the same Verifier into other memory or skill-library baselines to evaluate whether its gains specifically depend on the proposed skill mechanism. Combined with the reliability concerns in Points 1 and 2, stronger evidence is needed to justify the Verifier as a core novel contribution.

Weakness 4

For instance, Table 8 is overly wide, which disrupts the page layout and hinders readability; splitting the table or adjusting its column layout is recommended. In addition, the appendix includes extensive verbatim prompts and full JSON templates. While valuable for reproducibility, these materials could be streamlined by moving complete templates to the accompanying code repository and retaining only key input-output schemas, control rules, and illustrative examples in the paper to make the presentation more concise and accessible.

Questions for authors

See Critical Weaknesses for details.

7 comments
AuthorsOpenReview ↗

Response to Reviewer qTDZ (Part1/3)

We thank Reviewer qTDZ for the careful review. We answer Weaknesses 1–4 in 3 comments.

W1 — What the policy receives after a correction; resume vs. replan; post-hand-off behaviour

Why no rationale is passed. The corrective action is executed as the next environment action; the policy then sees the corrected state as an ordinary action–observation transition, and no verifier output ever enters its prompt (an escalation only re-issues the first-turn plan directive, row 3). The same-signal control (Table 15; ALFWorld held-out success %, 3 seeds) isolates this choice: the verifier, its context, its trigger rule and the code path are identical in the three verifier arms, and one flag selects what is done with the verdict.

Armseed 41seed 42seed 43mean ± stdΔ\Delta vs. row abovesign
no verifier54.4855.9756.7255.72 ± 1.14——
prompt-neutral (equal-length neutral text)55.9754.4857.4655.97 ± 1.49+0.25 ± 1.552/3
prompt-advice (verdict as advice)55.9757.4656.7256.72 ± 0.75+0.75 ± 1.971/3
direct execution (SKT)57.4658.9660.4558.96 ± 1.50+2.24 ± 1.293/3

Executing the verdict adds +2.24+2.24 pp, about 70% of the total +3.23+3.23 pp and the only increment positive on every seed.

Rationale and plan-synchronisation ablation (new). We added two additional arms to direct execution. The corrective action is executed as before; the arms differ only in one sentence appended to the next observation: the verifier's one-line reason (rationale), "Plan step kk is done; next plan step …\ldots" (plan-progress note)**

ConfigurationSeed 41Seed 42Seed 43Mean ± Std
Direct execution (SKT)57.4658.9660.4558.96 ± 1.50
+ Plan-progress note58.2161.1957.4658.95 ± 1.97
+ Verifier rationale58.2158.2154.4856.97 ± 2.15

Neither the plan-progress note nor the verifier rationale consistently improves upon direct execution. The plan-progress note achieves comparable performance with mixed effects across seeds. Adding the verifier rationale instead reduces average performance, despite a marginal improvement on seed 41 (+0.75 pp). Together, these results suggest that executing the verifier's corrective action is sufficient in this setting, while explicitly communicating the rationale or plan progress provides no consistent additional benefit.

Resume vs. replan. After an executed corrective, a verify-after-fix call returns now_landed (the plan step is now achieved), next_action (another corrective) and should_escalate_replan (the observation shows the plan is fundamentally wrong). The controller applies the first matching row. Table 14 (the events of the Critique Engine) in Appendix D.2 already gives the escalation ("Escalate": re-issue the first-turn plan directive, c←1c \leftarrow 1); Algorithm 1 does not show, so we clarify it here. The revision will add this table to Appendix D.2 and an "otherwise" case to Algorithm 1:

RowVerifier outputController action
1now_landed truec←c+1c \leftarrow c+1; control returns to the policy
2now_landed false, next_action givenqueue it; it is executed as the next environment action and followed by another verify-after-fix call
3now_landed false, no next_action, should_escalate_replan trueraise the replan flag, c←1c \leftarrow 1; the first-turn plan directive is re-issued at the next policy prompt
4otherwise (e.g. fix_worked true, now_landed false)cc unchanged; control returns to the policy

Replanning takes place within the same attempt: the environment is not reset, the action–observation history is kept, and every action, corrective or not, is charged to the budget TT.

Post-hand-off behaviour. We re-examined each of the 898 correction records of the three evaluation seeds for whether its targeted condition held afterwards: effective 294 (32.7%); redundant re-navigation to an already-satisfied state 258 (28.7%); command rejected by the environment 235 (26.2%); precondition unmet 87 (9.7%); misdirected (wrong object or place) 19 (2.1%); indeterminate 5 (0.6%). The three seeds rescue 4, 4 and 5 trials, and the end-to-end paired gain after all such costs is +3.23±0.43+3.23\pm0.43 pp (3/3 seeds) at about 3% wall-clock overhead. This accounting will be added into Section 4.5 as a paragraph "After a correction" in revision.

AuthorsOpenReview ↗

Response to Reviewer qTDZ (Part2/3)

W2 — Tracker, flawed plans, shared weights

Tracker and gate. The tracker is a deterministic verb/target matcher with no LLM call, and every boundary it proposes is re-checked by the verifier, whose verdict must be grounded in an excerpt of the latest observation. A spurious trigger therefore costs one verifier call, a missed boundary forgoes at most one correction, and every corrective action consumes one action of the budget TT. On a 300-step blinded, human-labelled sample, boundary precision is 97.99% and recall 96.05% (false-trigger rate 2.0%, 95% CI [0.7,5.8][0.7, 5.8]; missed-boundary rate 3.9%, [1.8,8.3][1.8, 8.3]), and the plan parser failed on 0 of 805 plans. The engine is a step-level realisation check; it does not revise the plan itself. The verifier observes about 36% of the plan steps, and widening this to 75% left success unchanged, so its reach is bounded by the policy's exploration.

Defective initial plans (new). We evaluate robustness to defective initial plans by introducing two deterministic perturbations (seeded by task and attempt): (1) omitting 25% of plan steps while retaining at least one, and (2) inserting an equal number of infeasible steps referencing nonexistent objects (e.g., "go to phantomdesk 9") at random positions. Only the initial plan of each attempt is corrupted, and the policy does not revisit its plan after the first step. Thus, the perturbations affect the plan observed by the tracker and verifier rather than directly modifying the policy's subsequent observations. All experiments use direct execution with frozen 140-task libraries, three attempts, and three seeds.

ConfigurationSeed 41Seed 42Seed 43Mean ± Std
No verifier54.4855.9756.7255.72 ± 1.14
Direct execution, original plan57.4658.9660.4558.96 ± 1.50
25% of plan steps omitted52.9957.4659.7056.72 ± 3.42
Infeasible steps inserted52.9964.9354.4557.46 ± 6.52

Both perturbations reduce mean success relative to the original plan, by 2.24 pp for step omission (56.72 ± 3.42 vs. 58.96 ± 1.50) and 1.50 pp for infeasible-step insertion (57.46 ± 6.52). However, their effects vary substantially across seeds, particularly for infeasible-step insertion (+5.97 pp on seed 42 versus −6.00 pp on seed 43). Both configurations still outperform the no-verifier baseline on average (+1.00 and +1.74 pp, respectively), although neither consistently does so across all seeds. This behavior is partly explained by the tracker's limited propagation of plan defects. For omitted steps, the plan pointer reaches the omission position in only 9.9% of attempts, while another 17% involve omissions confined to the plan tail. For inserted infeasible steps, the pointer reaches an inserted step in 29.8% of attempts but cannot advance because no action matches it, preventing subsequent verifier calls within that attempt. The system remains stable and robust under defective plans.

Shared weights. The verifier reads the plan, the current step, the last H=5H=5 action–observation pairs and the latest action. The benefit of the separate context is isolated in Table 15 and replicated on three backbones: the paired gain of the Critique Engine is +3.23±0.43+3.23\pm0.43 (Qwen2.5-7B), +2.74±1.13+2.74\pm1.13 (Qwen2.5-14B) and +2.24±0.75+2.24\pm0.75 pp (DeepSeek-V4-Flash API), positive in all nine backbone × seed cells.

Beyond average gains. Our paper reports the average scores of the ablations together with per-seed values and paired differences with sign counts, the correction taxonomy, cost and reach. The defective-plan runs above add direct evidence of task-level recovery: with an omitted or an infeasible plan, success stays at the level of the original plan and above no-verifier setting, and the engine does not enter a correction loop.

AuthorsOpenReview ↗

Response to Reviewer qTDZ (Part3/3)

W3 — Skill library, skill mechanisms, Verifier (1/2)

Share of the library. The Skill Engine is evaluated with the Critique Engine disabled: 55.72 ± 1.14 against 45.02 ± 4.11 for ReAct (+10.70±4.56+10.70\pm4.56 pp) and 49.25 ± 3.26 for SkillNet's 37 authored skills under the same retriever and kk (+6.47±4.40+6.47\pm4.40 pp), positive on 3/3 seeds each; the correction stage adds +3.23±0.43+3.23\pm0.43 pp (3/3 seeds). The claim is that each engine is positive on every seed, not that either depends on the other. The revision will state this in the abstract and the introduction:

[…] The two engines are separable, and we evaluate each against its own controls rather than claiming that either depends on the other. (Abstract) […] The two engines are separable: each is evaluated against its own controls, and we claim that each contributes on every seed, not that either depends on the other. (Introduction)

What is specific to the skill mechanisms. Removing one operator and rebuilding the library lowers success on all three seeds: without the Creator 48.01±0.8748.01\pm0.87 (−10.95±1.73-10.95\pm1.73), without the Distiller 53.23±3.5353.23\pm3.53 (−5.73±2.15-5.73\pm2.15), without the Refiner 54.98±1.5654.98\pm1.56 (−3.98±0.43-3.98\pm0.43), against 58.96±1.5058.96\pm1.50 for the full system; the best seed of each ablated configuration is below the worst seed of the full system. The revised Table 1 adds the closest methods, and the caption no longer claims uniqueness:

MethodExperienceFrom failureConsolidates failuresVerificationBackbone
AutoManualrule manualerror rulesrules merged above a cap—GPT-4-turbo
ReasoningBankreasoning memory itemsjudged failures as pitfallsfalse (append only)—Gemini-2.5, Claude-3.7
Mem$^p$procedural memoryrevised in place after a failed usefalse—GPT-4o, Claude, Qwen2.5-72B
SKT (ours)evolved NL procedurespatch + distilltruedirect actuationQwen2.5-7B-Instruct

Revised caption: "SKT combines natural-language procedural skills evolved from both success and failure, consolidation of failures across trials, and verification by direct actuation, with a frozen open-weight backbone; to our knowledge, none of the listed methods combines all three." Section 2 states the contrast: SKT differs in what is stored (procedures with ordered instructions and anti-patterns, retrieved per task rather than accumulated in context) and in how failure enters the store (ExpeL turns a failure into an insight, ReasoningBank into a memory item appended to the bank, Mem$^p$ into a revision of the entry that was used; SKT's refiner patches the specific procedure that misled the agent, and its distiller clusters many failures, with a support threshold, into a new procedure).

W3 — Verifier-only ablation, and the Verifier on other libraries (2/2)

We ran the suggested factorial design under the same collection and evaluation settings. We crossed the Critique Engine (off, on) with three library conditions: no library, the 37 authored SkillNet skills with our retriever (k=3k=3), and SKT's self-evolved library built from the 140 collection tasks. "Off" removes the entire critique engine.

Critique EngineLibraryseed 41seed 42seed 43mean ± std
offnone (ReAct)44.7849.2541.0445.02 ± 4.11
onnone (verifier only)47.7656.744.7849.75 ± 6.21
offSkillNet (37 authored)52.9947.7647.0149.25 ± 3.26
onSkillNet (37 authored)53.7350.7550.7551.74 ± 1.72
offSKT (Ours)54.4855.9756.7255.72 ± 1.14
onSKT (Ours)57.4658.9660.4558.96 ± 1.50

The Critique Engine consistently improves success across all three library conditions. Without any library, it increases success by +4.73 pp (45.02 → 49.75), with improvements on all three seeds, demonstrating that its benefits do not depend on our self-evolved library. With the authored SkillNet skills, it provides a +2.49 pp gain (49.25 → 51.74), again improving all three seeds. With SKT's self-evolved library, the gain reaches +3.24 pp (55.72 → 58.96), also positive across all seeds. With the Critique Engine enabled, SKT's library outperforms the authored SkillNet skills by +7.22 pp and the no-library configuration by +9.21 pp. The two engines are separable, and each is evaluated against its own controls.

W4 — Presentation: width of Table 8 and length of the appendix prompts

We thank the reviewer for these two presentation suggestions. We will fix tables with narrower column spacing in revision.

Appendix prompts. Appendix E prints the verbatim operator, verifier, plan-directive and policy prompts. In the revision, it will retain key input-output schemas, control rules, and illustrative examples in the paper to make the presentation more concise and accessibl; as the Reproducibility statement says, the prompts will be released together with the framework code.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.