- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces SPARK, which turns environment-verified agent trajectories into reusable skills. It also proposes PDI, a measure based on trajectory, plan, and memory similarity, with an online version that guides interventions. Experiments cover SkillsBench, generated task variants, and ALFWorld.
Critical strengths
(1) Using execution evidence and environment feedback is a practical way to improve the quality of distilled skills. (2) Repeated skill-generation experiments and evaluations on 300 task variants provide evidence beyond selected examples and simple correlations.
Critical weaknesses
(1) PDI rely on word-frequency similarity, which does not directly establish grounded execution. The intervention comparison also changes retry guidance, leaving the specific value of the PDI trigger unclear. (2) Main evaluations reuse source task instance, and some skills contain detailed solutions. The generated variants provide transfer evidence but largely preserve workflows and can be easier. Generalization to new task structures remains uncertain.
Questions for authors
(1) Does PDI outperform simple failure or stagnation triggers when retry messages and budgets are identical? (2) What task structure and solution content are shared between source tasks and the 300 variants? (3) Are ALFWorld teacher and evaluation tasks disjoint, and what sample count underlies each reported result?
5 comments
Questions
Q1. PDI vs. simple triggers
We thank the reviewer for this precise question. PDI's trigger is passive: it fires only when the run's own proxy falls below τ after a failed attempt (Sec. 3.2), so the timing is a diagnosis, not a schedule we set. To give an example: for a person who is not ill, a regimen involving either periodic or continuous medication would be misguided. The existing judgment rules are part of the effectiveness of PDI. The simple rules are already inside PDI: every retry follows a failure ("Failed attempts trigger a Reflect step", App. H.1), so a failure rule fires on stuck and progressing runs alike, and unchanged failing tests are one input of ossification, non-predictive on their own (App. B). During the rebuttal phase, we tested several methods for triggering PDI. A teacher model equipped with PDI successfully solved 69 problems. When PDI was triggered periodically or randomly, the success rate dropped to 48 and 43, respectively—to the extent that it was no longer possible to obtain a sufficient number of samples to evaluate the approach on student models.
Q2. What source tasks and variants share
We thank the reviewer for this question. They share the workflow (input type, processing steps, output contract), "while purposely resampling task-specific contents" (Sec. 4.4). Each variant has its own instruction, inputs, oracle and verifier, accepted only after its oracle passes and a critique checks for "answer leakage" (App. H.2). Because the inputs differ, the source answer does not carry over.
Q3. ALFWorld split and sample counts
We thank the reviewer for this question. Teacher exploration and student evaluation use the same 30 games, drawn from ALFWorld's eval_out_of_distribution split (134 games, App. M), as in the per-instance SkillsBench protocol. All three conditions run on all 30: Pick & Place 4, Clean 5, Heat 5, Cool 7, Examine 7, Pick Two 2. Copying cannot explain the gains: the GPT-5.5 teacher solved none of the Pick & Place (0/4) or Cool & Place (0/7) games, yet the student reaches 25.0% and 28.5% (App. M).
We hope our response has addressed the points you raised. We realize your time is valuable, and we would sincerely appreciate any further questions or suggestions.
Weaknesses
W1. Word-frequency similarity; retry guidance in the intervention
We greatly thank the reviewer for this incisive comment.
By design, the skill is written only after a verified success, from that run's commands, tests and output (Sec. 3.1.1), and the memo is rewritten after each failure from the actual commands and test results (Sec. 3.1.2). PDI then checks which the skill follows: verified commands or the memo's plan. Shared words are not the signal: plain word overlap with the successful commands or the memo is non-predictive (App. B). On
lean4-proof, two skills with shared vocabulary (Jaccard 0.37) are separated by grounding (0.62 vs. 0.78), and students succeed with the higher one: 15 commands/3 failures → 5/0 (App. D).Both arms run the same pipeline and retry prompt after every failure. When PDI fires, it adds only "prompt-only guidance encouraging broader hypothesis revision", with no task knowledge; the memo and its Verified Facts stay; the strong level withholds only the Next Strategy (Sec. 3.2). What PDI decides is when to intervene: once a run stagnates. Every retry follows a failure, so a failure rule cannot tell stuck runs from progressing ones; unchanged failing tests are already part of ossification and alone are non-predictive (App. B). PDI picks the right runs: replayed without intervention, flagged runs' skills fix 16% of student failures vs. 37% unflagged. Acting on its flags lifts the pass rate from 9.3% to 28.1% under matched compute (p<0.001, Table 2).
W2. Same-instance evaluation, detailed skills, and variants
We sincerely thank the reviewer for this thoughtful point.
This is the SkillsBench protocol, where skills are "in-context procedural documents that encode task-specific knowledge such as execution ordering, environment constraints, and verifier contracts" (Sec. 1). Skill generation is evaluated the same way: SkillsBench's self-generated condition uses "two isolated sessions per task", where a creator writes the skill and "a fresh agent then runs the task exactly as in the curated-Skills condition" (Li et al., 2026); CoEvoSkills (COLM 2026) likewise evolves and scores a skill per SkillsBench task on that task's oracle. Where a skill is detailed, as on
lean4-proof, it "transfers an execution recipe anchored in verified evidence" (App. D): verifier flags, tactics, test in /tmp first. Instance content is normal in SkillsBench: its own human skill forlab-unit-harmonizationships the reference ranges its solution uses. What matters is surviving new instances: a recorded answer does not, a recorded method does. On 300 variants with resampled contents, PDI skills reach 80.6% vs. 33.7% without skills (Table 2).A shared workflow is what makes a skill reusable. If the tools, their order and their functions all change, it is a different task, testing base ability rather than skill transfer. The pool measures generalization, not difficulty: "both arms face the identical pool, so it is their contrast that carries the claim" (Sec. 4.4). PDI skills reach 80.6% vs. 67.2% without PDI, on a pool that is not uniformly easy (GPT-5.4-mini baseline: 7.0%). For a new task structure, ALFWorld uses "discrete text commands rather than shell operations" (App. M). There, PDI skills raise success from 16.7% to 40.0%, including on task types the teacher never solved, where there was no answer to copy (App. M). We further ran SPARK on τ-bench retail (Yao et al., 2024), a customer-service dialogue domain judged on the final database state: GPT-6.1-sol explores 15 source tasks, and GPT-5.6-luna is evaluated only on 35 disjoint held-out tasks, so no source instance is reused.
Why PDI helps, from one exploration run (τ-bench airline, a cancellation request): after two failures with the same wrong reservation record, the teacher's new plan largely repeated the old one, so PDI fell below τ and fired a soft intervention. Within the next two attempts the plan changed direction, from guessing which booking to cancel to asking the customer for the booking and reason before changing anything, and the run succeeded. A pass/fail signal sees only repeated failures; PDI sees that the plan itself has stopped changing.