acceptodds
Under review as a conference paper at ICLR 2027

Evidence Over Plans: Online Trajectory Verification for Skill Distillation

Abstract

Agent skills can remarkably improve task success rates by using human-written procedural documents, but their quality is difficult to assess without environment-grounded verification. Existing skill generation methods heavily rely on preference logs rather than direct environment interaction, often yielding negligible or even degraded gains. We identify that this reflects a fundamental timing problem: robust skills should be *posterior-based*, distilled from environment interaction rather than composed from offline plans. In this study, we introduce the **Posterior Distillation Index (PDI)**, a trajectory-level metric that quantifies whether a distilled skill reflects environment-verified execution evidence rather than offline planning. To operationalize PDI, we present **SPARK** (**S**tructured **P**ipelines for **A**utonomous **R**unnable tas**K**s and s**K**ill generation), a closed-loop framework for automated agent skill synthesis. SPARK generates environment-grounded trajectories from which PDI is computed, and it uses PDI as an online intervention signal to improve skill generation since its creation time. Critically, SPARK enables trajectory-level analysis for skill generation by preserving key execution logs, verifier signals, and memo histories. Across 86 runnable tasks, SPARK-generated skills consistently surpass no-skill baselines and outperform human-written skills on major student models. These findings demonstrate that key posterior evidence distillation enables transferable skill generation via environment interaction. We release anonymous code in [https://anonymous.4open.science/r/spark_anonymous-2C82/](https://anonymous.4open.science/r/spark_anonymous-2C82/).

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agents, Agent Skill Learning, Automated Skill Synthesis

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_PfeVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SPARK, which turns environment-verified agent trajectories into reusable skills. It also proposes PDI, a measure based on trajectory, plan, and memory similarity, with an online version that guides interventions. Experiments cover SkillsBench, generated task variants, and ALFWorld.

Critical strengths

(1) Using execution evidence and environment feedback is a practical way to improve the quality of distilled skills. (2) Repeated skill-generation experiments and evaluations on 300 task variants provide evidence beyond selected examples and simple correlations.

Critical weaknesses

(1) PDI rely on word-frequency similarity, which does not directly establish grounded execution. The intervention comparison also changes retry guidance, leaving the specific value of the PDI trigger unclear. (2) Main evaluations reuse source task instance, and some skills contain detailed solutions. The generated variants provide transfer evidence but largely preserve workflows and can be easier. Generalization to new task structures remains uncertain.

Questions for authors

(1) Does PDI outperform simple failure or stagnation triggers when retry messages and budgets are identical? (2) What task structure and solution content are shared between source tasks and the 300 variants? (3) Are ALFWorld teacher and evaluation tasks disjoint, and what sample count underlies each reported result?

5 comments

Weaknesses

W1. Word-frequency similarity; retry guidance in the intervention

We greatly thank the reviewer for this incisive comment.

PDI rely on word-frequency similarity, which does not directly establish grounded execution.

By design, the skill is written only after a verified success, from that run's commands, tests and output (Sec. 3.1.1), and the memo is rewritten after each failure from the actual commands and test results (Sec. 3.1.2). PDI then checks which the skill follows: verified commands or the memo's plan. Shared words are not the signal: plain word overlap with the successful commands or the memo is non-predictive (App. B). On lean4-proof, two skills with shared vocabulary (Jaccard 0.37) are separated by grounding (0.62 vs. 0.78), and students succeed with the higher one: 15 commands/3 failures → 5/0 (App. D).

The intervention comparison also changes retry guidance, leaving the specific value of the PDI trigger unclear.

Both arms run the same pipeline and retry prompt after every failure. When PDI fires, it adds only "prompt-only guidance encouraging broader hypothesis revision", with no task knowledge; the memo and its Verified Facts stay; the strong level withholds only the Next Strategy (Sec. 3.2). What PDI decides is when to intervene: once a run stagnates. Every retry follows a failure, so a failure rule cannot tell stuck runs from progressing ones; unchanged failing tests are already part of ossification and alone are non-predictive (App. B). PDI picks the right runs: replayed without intervention, flagged runs' skills fix 16% of student failures vs. 37% unflagged. Acting on its flags lifts the pass rate from 9.3% to 28.1% under matched compute (p<0.001p<0.001, Table 2).

W2. Same-instance evaluation, detailed skills, and variants

We sincerely thank the reviewer for this thoughtful point.

Main evaluations reuse source task instance, and some skills contain detailed solutions.

This is the SkillsBench protocol, where skills are "in-context procedural documents that encode task-specific knowledge such as execution ordering, environment constraints, and verifier contracts" (Sec. 1). Skill generation is evaluated the same way: SkillsBench's self-generated condition uses "two isolated sessions per task", where a creator writes the skill and "a fresh agent then runs the task exactly as in the curated-Skills condition" (Li et al., 2026); CoEvoSkills (COLM 2026) likewise evolves and scores a skill per SkillsBench task on that task's oracle. Where a skill is detailed, as on lean4-proof, it "transfers an execution recipe anchored in verified evidence" (App. D): verifier flags, tactics, test in /tmp first. Instance content is normal in SkillsBench: its own human skill for lab-unit-harmonization ships the reference ranges its solution uses. What matters is surviving new instances: a recorded answer does not, a recorded method does. On 300 variants with resampled contents, PDI skills reach 80.6% vs. 33.7% without skills (Table 2).

The generated variants ... largely preserve workflows and can be easier. Generalization to new task structures remains uncertain.

A shared workflow is what makes a skill reusable. If the tools, their order and their functions all change, it is a different task, testing base ability rather than skill transfer. The pool measures generalization, not difficulty: "both arms face the identical pool, so it is their contrast that carries the claim" (Sec. 4.4). PDI skills reach 80.6% vs. 67.2% without PDI, on a pool that is not uniformly easy (GPT-5.4-mini baseline: 7.0%). For a new task structure, ALFWorld uses "discrete text commands rather than shell operations" (App. M). There, PDI skills raise success from 16.7% to 40.0%, including on task types the teacher never solved, where there was no answer to copy (App. M). We further ran SPARK on τ-bench retail (Yao et al., 2024), a customer-service dialogue domain judged on the final database state: GPT-6.1-sol explores 15 source tasks, and GPT-5.6-luna is evaluated only on 35 disjoint held-out tasks, so no source instance is reused.

τ-bench retail (35 held-out tasks)No skillSkill w/o PDISkill w/ PDI
Success rate68.6%71.4%80.0%

Why PDI helps, from one exploration run (τ-bench airline, a cancellation request): after two failures with the same wrong reservation record, the teacher's new plan largely repeated the old one, so PDI fell below τ and fired a soft intervention. Within the next two attempts the plan changed direction, from guessing which booking to cancel to asking the customer for the booking and reason before changing anything, and the run succeeded. A pass/fail signal sees only repeated failures; PDI sees that the plan itself has stopped changing.

Questions

Q1. PDI vs. simple triggers

We thank the reviewer for this precise question. PDI's trigger is passive: it fires only when the run's own proxy falls below τ after a failed attempt (Sec. 3.2), so the timing is a diagnosis, not a schedule we set. To give an example: for a person who is not ill, a regimen involving either periodic or continuous medication would be misguided. The existing judgment rules are part of the effectiveness of PDI. The simple rules are already inside PDI: every retry follows a failure ("Failed attempts trigger a Reflect step", App. H.1), so a failure rule fires on stuck and progressing runs alike, and unchanged failing tests are one input of ossification, non-predictive on their own (App. B). During the rebuttal phase, we tested several methods for triggering PDI. A teacher model equipped with PDI successfully solved 69 problems. When PDI was triggered periodically or randomly, the success rate dropped to 48 and 43, respectively—to the extent that it was no longer possible to obtain a sufficient number of samples to evaluate the approach on student models.

Q2. What source tasks and variants share

We thank the reviewer for this question. They share the workflow (input type, processing steps, output contract), "while purposely resampling task-specific contents" (Sec. 4.4). Each variant has its own instruction, inputs, oracle and verifier, accepted only after its oracle passes and a critique checks for "answer leakage" (App. H.2). Because the inputs differ, the source answer does not carry over.

Q3. ALFWorld split and sample counts

We thank the reviewer for this question. Teacher exploration and student evaluation use the same 30 games, drawn from ALFWorld's eval_out_of_distribution split (134 games, App. M), as in the per-instance SkillsBench protocol. All three conditions run on all 30: Pick & Place 4, Clean 5, Heat 5, Cool 7, Examine 7, Pick Two 2. Copying cannot explain the gains: the GPT-5.5 teacher solved none of the Pick & Place (0/4) or Cool & Place (0/7) games, yet the student reaches 25.0% and 28.5% (App. M).

We hope our response has addressed the points you raised. We realize your time is valuable, and we would sincerely appreciate any further questions or suggestions.

Reviewer_WGiaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how environment-interaction trajectories can be distilled into reusable agent skills. It introduces SPARK, a pipeline that collects execution traces, maintains exploration memos, and generates SKILL.md documents after successful execution. Its central contribution is the Posterior Distillation Index (PDI), which combines execution grounding, plan copying, and memo ossification using token-distribution similarities. An online proxy triggers prompt interventions intended to redirect stagnant exploration. Experiments evaluate downstream skill utility on SkillsBench, generated task variants, and ALFWorld.

Critical strengths
  1. The paper evaluates skill utility through execution. It measures downstream task performance rather than relying exclusively on an LLM’s judgment of document quality. Preserving commands, verifier feedback, and memo histories also creates useful material for analyzing skill generation.

  2. The predictive analysis provides preliminary support for PDI. The reported correlation with skill gain, task-level high/low-PDI comparison, and sensitivity analyses indicate that the score contains information about downstream utility. These results should not be dismissed simply because the metric is heuristic.

  3. The intervention evaluation goes beyond retrospective correlation. Regenerating both experimental arms over three seeds under matched compute is a useful design choice. The generated-variant and ALFWorld evaluations provide additional evidence that the overall approach can be useful beyond the original task-instance evaluation.

Critical weaknesses
  1. PDI’s construct validity is insufficiently established. The metric operationalizes evidence grounding, plan copying, and ossification through token-frequency similarities. These similarities do not directly establish whether a skill’s claims are supported by execution evidence. A correct plan may appropriately resemble both the executed solution and the final skill, while verified facts may remain stable during productive exploration. Conversely, lexical changes need not represent substantive learning. The reported association with student performance supports predictive usefulness, but does not validate the proposed interpretations of the components. Independent annotations of evidence support, or controlled examples separating lexical similarity from semantic correctness, would substantially strengthen this claim.

  2. The intervention ablation does not isolate the value of PDI as a trigger. Table 2 shows a substantial improvement on the selected rerun subset: pooled pass rate increases from 9.3% to 28.1%. However, the treatment combines metric-based triggering with instructions to reconsider hypotheses and, under stronger intervention, removal of the previous strategy. The control lacks this intervention package. Consequently, the result establishes the usefulness of the package relative to no intervention, but does not determine whether PDI selects better intervention times than consecutive-failure rules, unchanged-test detection, periodic intervention, or random intervention at a matched frequency. This missing comparison is central because the metric is the main claimed contribution.

  3. The offline metric and online proxy measure different relationships. Offline execution grounding compares the final skill with successful execution commands, whereas the online proxy compares current commands with the memo’s verified facts. Offline plan copying compares the final skill with accumulated plans, whereas its online counterpart compares consecutive plans. These substitutions are understandable operationally, but they require separate validation. The correlation between completed-trajectory PDI and skill utility does not establish that low online-proxy values predict poor eventual skills or identify moments when intervention is beneficial. The case studies illustrate behavior but do not close this gap.

  4. The predictive and component analyses do not establish sufficient incremental value. The reported pooled correlation of 0.364 and average held-out correlation of 0.289 suggest useful but limited predictive information. The paper should compare the complete index directly against individual components and simple features such as repeated failures, reward progress, and document length under the same task-held-out protocol. Appendix F describes component-removal results without providing sufficiently detailed numerical results and uncertainty estimates. Sensitivity to weights supports some robustness, but does not establish that all three components or the proposed combination are needed.

  5. The evaluation supports narrower conclusions than some of the framing suggests. The main benchmark transfers a skill from teacher exploration to a student solving the same task instance. This can demonstrate useful solution transfer, but provides limited evidence of reusable skill acquisition. The Lean case, where the refined skill contains a complete reference proof and becomes substantially longer, illustrates the need to distinguish general procedural knowledge from detailed solution provision. The generated variants partially address this concern, but preserve the source workflow and are acknowledged to be easier. Likewise, the targeted low-proxy-PDI rerun subset establishes efficacy for that selected population, rather than broad benefit or absence of adverse effects across the entire task suite.

Questions for authors
  1. How does PDI-triggered intervention compare with consecutive-failure, unchanged-test, periodic, and random triggers when intervention content, frequency, and compute are matched?
  2. Does the online proxy predict eventual offline PDI or downstream skill utility on held-out tasks? Does it identify intervention opportunities better than simple failure signals?
  3. Can the authors provide numerical component-ablation results, uncertainty estimates, and comparisons against individual components and basic trajectory features?
  4. Can PDI distinguish semantically different skills with similar vocabulary, and recognize correctly preserved plans or stable verified facts?
  5. How many tasks enter each predictive analysis and rerun subset, and what happens when the intervention policy is evaluated on the complete task suite?
5 comments

W1. Construct validity of the components

We greatly thank the reviewer for this incisive point.

These similarities do not directly establish whether a skill's claims are supported by execution evidence.

The evidence comes from what PDI compares against. The skill is written only after a verified success, from that run's commands and tests (Sec. 3.1.1); the memo is rewritten from actual results after each failure (Sec. 3.1.2). Execution grounding compares the skill with verifier-passed commands, and ossification uses "the distributional persistence of failed test sets" (Sec. 3.2). PDI thus asks whether the skill follows what was verified or the plan written beforehand. Shared words are not the signal: plain skill↔command and skill↔memo overlap are non-predictive (App. B), while PDI reaches ρ = +0.364.

A correct plan may appropriately resemble both the executed solution and the final skill

A correct plan is not penalized: once executed and passed, it is in the successful commands, which grounding rewards. What lowers PDI is a skill that mirrors the plan more than the execution, and it hurts students: with grounding held high, more plan copying lowers Δr from +0.377 to +0.143 (Table 1).

verified facts may remain stable during productive exploration

Stable facts alone do not drive ossification. It averages fact stability with failed-test persistence, so it rises when the facts stay the same while the same tests keep failing. Even then nothing verified is lost: the Verified Facts stay, and only the stale Next Strategy is withheld (Sec. 3.2). Such runs do hurt transfer: "convergent-trajectory skills even produce negative Δr\Delta r" (Sec. 4.2.3).

lexical changes need not represent substantive learning

PDI does not reward raw change: pure-change features (3-gram novelty, info gain, lexical density) are non-predictive negative controls (App. B).

… controlled examples separating lexical similarity from semantic correctness

App. D is such a controlled example. On lean4-proof, two skills share much vocabulary (token Jaccard 0.37), yet grounding separates them (0.62 vs. 0.78), and students succeed with the higher one: 15 commands/3 failures → 5/0, and FAIL → PASS on GPT-5.1-Codex and GLM-4.5-Air. Seven students executing each skill also check its content.

W2. PDI as a trigger

We greatly thank the reviewer for this insightful comment.

the treatment combines metric-based triggering with instructions … and … removal of the previous strategy

Trigger and instruction are one treatment: PDI is the diagnosis and the instruction is its prescription. The soft level is "prompt-only guidance encouraging broader hypothesis revision"; only "if two consecutive steps both trigger" does it withhold the previous Next Strategy and ask the agent "to anchor on Verified Facts and Current Error Pattern rather than continuing the stale plan" (Sec. 3.2). Neither adds task knowledge.

whether PDI selects better intervention times than consecutive-failure rules, unchanged-test detection, periodic intervention, or random intervention at a matched frequency

PDI's trigger is passive: we do not choose when it fires. It fires only when the run's own proxy falls below τ after a failed attempt (Sec. 3.2). A doctor gives medicine when a test shows the illness; giving it on a schedule (periodic) or by lottery (random) ignores the diagnosis and can only harm a healthy person. The healthy are runs making progress.

The other two rules are already inside PDI's diagnosis. Every monitored step follows a failure ("Failed attempts trigger a Reflect step", App. H.1), so a failure rule fires on stuck and progressing runs alike. Unchanged tests are one input of ossification, non-predictive alone (test-stuck ratio, App. B).

W3. Offline index vs. online proxy

We greatly thank the reviewer for this careful point.

they require separate validation

We ran it, replaying the unchanged proxy observe-only over the original non-intervened trajectories the seven students evaluate (Sec. 4.2). Average proxy-PDI tracks final PDI: ρ = 0.79 (p=1.7×10−7p = 1.7\times10^{-7}). The proxy has no fitted parameters. We will add this to the appendix.

does not establish that low online-proxy values predict poor eventual skills

They do. On the same replay, proxy-PDI correlates with student skill gain (ρ = +0.25, p=7.2×10−4p = 7.2\times10^{-4}). Skills from runs the proxy would flag turn 16% of student failures into passes, vs. 37% for unflagged runs (p=7.6×10−4p = 7.6\times10^{-4}).

or identify moments when intervention is beneficial. The case studies illustrate behavior but do not close this gap.

Timing is tested in Table 2, not the case studies (App. C): intervening at flagged steps lifts pooled pass rate from 9.3% to 28.1% over three matched-compute seeds (p<0.001p<0.001), on all five students.

W4. Incremental value

We thank the reviewer for this constructive suggestion.

useful but limited predictive information

Students disagree on which skills help ("off-diagonal rates are mostly moderate ($<$60%)", Sec. 4.2.1), capping any per-skill score.

compare the complete index … against individual components and simple features … under the same task-held-out protocol

App. F's 5-fold task-held-out protocol, 200 random splits (mean ± std):

PredictorHeld-out ρPass rate
PDI0.315 ± 0.08150.8%
Grounding only0.258 ± 0.07648.8%
Plan copying only0.212 ± 0.07847.5%
Ossification only0.207 ± 0.07649.2%
Repeated failures0.157 ± 0.07145.8%
Reward progress−0.128 ± 0.17240.7%
Document length0.158 ± 0.07544.1%

Controlling for repeated failures or length, PDI's partial ρ stays at 0.24–0.33.

does not establish that all three components or the proposed combination are needed

Each component is significant alone (ρ = 0.227–0.302, p≤0.002p \le 0.002) yet below the full index, and catches a different failure: grounding checks whether the skill's actions have execution evidence, plan copying catches a strategy restated without change, and ossification catches repeated failures without new verified facts.

W5. Scope of the conclusions

We thank the reviewer for this thoughtful point.

a student solving the same task instance … limited evidence of reusable skill acquisition

We further ran SPARK on τ-bench retail (Yao et al., 2024), a customer-service dialogue domain judged on the final database state: GPT-6.1-sol explores 15 source tasks, and GPT-5.6-luna is evaluated only on 35 disjoint held-out tasks, so no source instance is reused.

τ-bench retail (35 held-out tasks)No skillSkill w/o PDISkill w/ PDI
Success rate68.6%71.4%80.0%

A task type the teacher never solved leaves no answer to copy either: on ALFWorld, PDI skills help "even on task types where the teacher itself never succeeded during exploration" (Pick & Place 0% → 25.0%, Cool & Place 0% → 28.5%; App. M).

The Lean case … contains a complete reference proof and becomes substantially longer

The Lean skill "transfers an execution recipe anchored in verified evidence" (App. D): the verification command, the tactic chain, testing in /tmp first. A recorded answer does not survive resampled contents; a method does (+13.4 points on 300 variants, Table 2). Length is not the lever: the human Lean skills are longer (598 vs. 393 lines) yet pass 0 of 4 students.

The generated variants … preserve the source workflow and are acknowledged to be easier

A shared workflow is what makes a skill reusable, while task contents are resampled (Sec. 4.4). Easiness applies to every arm, as "both arms face the identical pool" (Sec. 4.4). PDI skills reach 80.6% with PDI vs. 67.2% without and 65.7% with human skills (GPT-5.4-mini baseline: 7.0%).

the targeted low-proxy-PDI rerun subset establishes efficacy for that selected population, rather than broad benefit or absence of adverse effects

The subset is where PDI acts: it excludes tasks "where the teacher agent succeeded in the first round and those where the PDI monitors remain within healthy thresholds that will not trigger interference" (Sec. 4.3). Elsewhere PDI never fires, so the arms are identical. On all 86 tasks, PDI-guided SPARK improves 21 and degrades 3 (AutoRefine 19/9, App. K); on ALFWorld, with no task selection, PDI wins under both teachers (40.0% vs. 30.0%; 30.0% vs. 23.3%; App. M).

Questions

Q1. PDI vs. other triggers at matched content, frequency and compute

We thank the reviewer for this precise question. PDI fires only when a run's own proxy drops below τ after a failure (Sec. 3.2), a diagnosis rather than a schedule. To take an analogy: for a person who is not ill, any regimen, whether taking medication periodically, continuously, or at random, would be misguided. The existing set of decision rules serves as one component of how PDI is implemented. The existing judgment rules are part of the effectiveness of PDI. During the rebuttal phase, we tested several methods for triggering PDI. A teacher model equipped with PDI successfully solved 69 problems. When PDI was triggered periodically or randomly, the success rate dropped to 48 and 43, respectively, to the extent that it was no longer possible to obtain a sufficient number of samples to evaluate the approach on student models.

Q2. Does the online proxy predict eventual PDI and skill utility?

We thank the reviewer for this direct question. Yes. Replayed observe-only, the proxy tracks final PDI (ρ = 0.79) and student skill gain (ρ = +0.25); flagged runs' skills fix 16% of student failures vs. 37% unflagged, and acting on its flags lifts pass rate from 9.3% to 28.1% (Table 2).

Q3. Numerical component ablation and simple features

We thank the reviewer for this request. Under App. F's task-held-out protocol, PDI (ρ = 0.315) beats each component (0.207–0.258), repeated failures (0.257), reward progress (−0.128) and length (0.158).

Q4. Similar vocabulary; preserved plans; stable facts

We thank the reviewer for this question. (i) Yes: on lean4-proof, grounding separates two lexically similar skills (0.62 vs. 0.78), and students succeed with the higher one (App. D). (ii) An executed correct plan appears in the verified commands, which grounding rewards (Table 1). (iii) Ossification is high when stable facts coincide with the same tests failing again.

Q5. Counts, and the complete task suite

We thank the reviewer for this question. The predictive analysis uses all 30 iterative tasks (181 task–student pairs). The rerun subset holds the tasks the teacher did not solve in round one and on which PDI fires (Sec. 4.3); elsewhere both arms are identical. On the full 86-task suite, PDI-guided SPARK improves 21 tasks and degrades 3 (App. K).

Reviewer_oUrEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary
  1. Novel posterior-evidence framework: PDI metric directly quantifies whether skills are grounded in environment interaction vs. offline planning, addressing a fundamental gap in skill distillation.

  2. Strong empirical validation: Consistent improvements across 86 runnable tasks with practical SPARK pipeline implementation showing skill quality gains and transferability. Proposes online trajectory verification to improve skill distillation by distinguishing successful execution paths from failed ones. Evaluates evidence (actual outcomes) rather than relying on plan-based heuristics. Demonstrates benefits across multiple benchmarks with improved skill quality and reduced distillation costs.

Critical strengths

Novel posterior-evidence framework: PDI metric directly quantifies whether skills are grounded in environment interaction vs. offline planning, addressing a fundamental gap in skill distillation.

Strong empirical validation: Consistent improvements across 86 runnable tasks with practical SPARK pipeline implementation showing skill quality gains and transferability.

Critical weaknesses
  1. Limited scope of evaluation: All experiments on simulated/semi-simulated environments; unclear generalization to diverse real-world domains with different skill discovery dynamics.

  2. Insufficient ablation and baseline comparisons: PDI design choices, SPARK components, and comparison with simpler evidence-grounding alternatives not fully analyzed; cost-benefit analysis of trajectory verification overhead missing.

Questions for authors

How does PDI perform in domains where "successful execution" is ambiguous or multi-faceted (e.g., dialogue, long-horizon planning)? What is the computational overhead of SPARK's online trajectory verification vs. offline baselines?

Can you ablate PDI components (e.g., verifier signals, memo histories) to show individual contribution? How does performance scale with skill library size and complexity?

4 comments

Weaknesses

W1. Evaluation limited to simulated/semi-simulated environments; generalization to real-world domains unclear

We greatly thank the reviewer for raising the question of deployment scope.

All experiments on simulated/semi-simulated environments

SkillsBench is not a simulator: "86 tasks spanning 11 domains … each paired with … a Docker execution environment, and a deterministic pytest oracle" (Sec. 4). The released environments (anonymous repo, tasks/) include real failed CI builds from BugSwarm (fix-build-google-auto), the OSS-Fuzz builder, the Suricata IDS on a packet capture, and the Lean 4 toolchain, with real xlsx/pdf/pptx/pcap/STL inputs. For a terminal agent this is the deployment environment itself.

unclear generalization to diverse real-world domains

The main benchmark already spans "11 domains (e.g., Software Engineering, Finance, and Cybersecurity)" (Sec. 4). We further ran SPARK on τ-bench retail (Yao et al., 2024), a customer-service domain where the agent serves a simulated customer through tool calls and is judged on the final database state: GPT-6.1-sol explores 15 source tasks, and GPT-5.6-luna is evaluated on 35 disjoint held-out tasks.

τ-bench retail (35 held-out tasks)No skillSkill w/o PDISkill w/ PDI
Success rate68.6%71.4%80.0%

Why PDI helps, from one exploration run (τ-bench airline, a cancellation request): after two failures with the same wrong reservation record, the teacher's new plan largely repeated the old one, so PDI fell below τ and fired a soft intervention. Within the next two attempts the plan changed direction, from guessing which booking to cancel to asking the customer for the booking and reason before changing anything, and the run succeeded. A pass/fail signal sees only repeated failures; PDI sees that the plan itself has stopped changing.

different skill discovery dynamics

ALFWorld is such a domain: "actions are discrete text commands rather than shell operations" (App. M). Under both teachers, PDI-guided skills beat non-PDI skills (40.0% vs. 30.0%; 30.0% vs. 23.3%; no-skill 16.7%).

W2. Ablations, baselines, and verification overhead

We sincerely thank the reviewer for the detailed list.

PDI design choices

Smoothing α is swept in App. E and weights in App. F (equal weights win or tie on all 5 folds). We also compared the full index with each component under App. F's task-held-out protocol over 200 random splits (181 pairs, 30 tasks; held-out Spearman ρ, mean ± std):

PredictorHeld-out ρPass rate
PDI0.315 ± 0.08150.8%
Grounding only0.258 ± 0.07648.8%
Plan copying only0.212 ± 0.07847.5%
Ossification only0.207 ± 0.07649.2%

The full index scores above each of its components.

SPARK components

Table 1 separates the components (low plan copying + high grounding: Δr = +0.377; the reverse: +0.028). Online PDI is ablated over 3 matched-compute seeds: 9.3% → 28.1% (p<0.001p<0.001, Table 2). Fig. 5(a) ablates interaction itself: high-PDI skills beat interaction-free skills on all seven students.

simpler evidence-grounding alternatives

App. B tests 23 simple features, including plain overlap baselines (skill↔command overlap, skill↔memo overlap, unchanged-test ratio); all three are non-predictive, while PDI reaches ρ = +0.364 (p=4.7×10−7p = 4.7\times10^{-7}). On the same trajectories SPARK beats Trace2Skill, AutoRefine and EvoSkill (ΔMR +0.291 vs. ≤ +0.172, App. K), and it also beats GEPA and ACE (App. L).

cost-benefit of trajectory verification overhead

Verification adds no runs: it is the task's own oracle on attempts the teacher makes anyway. PDI is token statistics over recorded text, with no LLM or environment call, so Table 2's arms are compute-matched. The one-time teacher cost ($9.90–$24.72/task) amortizes over $0.02–$0.18/task students (App. G), and GPT-5.4-nano with SPARK skills (0.41) exceeds teacher Claude Opus 4.6 without skills (0.37).

Questions

Q1. Ambiguous success signals; overhead vs. offline baselines

We thank the reviewer for this insightful question.

domains where "successful execution" is ambiguous or multi-faceted

PDI does not require a binary success signal. Its inputs are the agent's actions, the memo, and whatever feedback the environment returns, and the oracle already gives graded rewards: "a reward r∈[0,1]r \in [0,1], where r=1r{=}1 indicates full task completion" (Sec. 4). ALFWorld is a long-horizon, multi-step domain, and PDI-guided skills raise success "even on task types where the teacher itself never succeeded during exploration", e.g., Pick & Place 0% → 25% (App. M). PDI therefore extracts usable evidence from partial successes.

computational overhead … vs. offline baselines

SPARK adds no extra environment runs for verification: the oracle that checks each attempt is the task's own test suite, run on attempts the teacher makes anyway. PDI itself is token statistics over text already recorded (commands, test output, memo), with no LLM call and no environment call, so its online cost is negligible next to the teacher's attempts. Table 2's two arms are therefore regenerated under matched compute. The one-time teacher cost ($9.90–$24.72 per task) is then reused by every student, whose runs cost $0.02–$0.18 per task (App. G).

Q2. Ablate PDI components (verifier signals, memo histories); scaling with skill library size and complexity

We greatly thank the reviewer for the suggestion.

ablate PDI components

Under App. F's task-held-out protocol (200 random splits, mean ± std), the full index reaches held-out ρ = 0.315 ± 0.081. The verifier signal alone (execution grounding) reaches 0.258 ± 0.076, and the memo history alone (memo ossification) reaches 0.207 ± 0.076, so each carries signal and the full index is above both. Memo history also has its own effect: ossification tracks the gap to human-written skills (ρ = −0.277, p=1.6×10−4p = 1.6\times10^{-4}, Sec. 4.2.4).

scale with skill library size and complexity

The library covers 86 tasks over 11 domains; evaluated on 300 variants, SPARK skills still improve every student (pooled 33.7% → 80.6% with PDI, Table 2). For complexity, Fig. 4 shows that over-compressing long trajectories into a short skill lowers skill gain for most students.

We hope our response has addressed your concerns. We know your time is valuable, and we would be very grateful for any further questions or suggestions.

AuthorsOpenReview ↗

Thank You for Your Valuable Comment :) !

As we approach the end of the response period, I would like to sincerely thank you for your valuable suggestions, which have helped us further improve the clarity of our paper. We genuinely hope that our responses have adequately addressed your concerns. If time permits and you have any further questions, we would be more than happy to respond promptly. Regardless, we wish you a wonderful day!

Comments on the paper

1 comment
AuthorsOpenReview ↗

With Gratitude to the AC and Reviewers: Summary of the Discussion

We thank the AC and all three reviewers for their careful reviews. We answered each point by quoting it, citing the submitted paper wherever it already holds the answer, and added four analyses during the rebuttal, summarized below.

1. Does PDI measure execution grounding, or only word overlap? (WGia W1, PfeV W1)

Shared words are not the signal: plain word overlap between the skill and the successful commands or the memo does not predict skill utility, while PDI does (App. B). PDI compares the skill with the commands that passed the verifier and with the plan written beforehand, and asks which of the two the skill follows. On lean4-proof, it separates two skills with similar vocabulary, and students succeed with the one it scores higher (App. D).

2. Are all three components needed? (WGia W4, oUrE W2/Q2)

We added the requested comparison under the paper's task-held-out protocol. The full index predicts skill utility better than any single component and better than simple features such as repeated failures, reward progress and document length.

3. Is PDI a useful trigger? (WGia W2/W3/Q1, PfeV W1/Q1)

PDI fires only when a run's own proxy drops after a failure (Sec. 3.2), and the intervention adds no task knowledge. In a new comparison with the same intervention, PDI triggering lets the teacher solve 69 tasks, against 48 for periodic and 43 for random triggering. Replayed observe-only on the original trajectories, the online proxy also tracks the final index and student skill gain. Acting on its flags lifts the pooled pass rate from 9.3% to 28.1% over three matched-compute seeds (Table 2), and on all 86 tasks PDI-guided SPARK improves 21 tasks while degrading 3 (App. K).

4. Same-instance evaluation and generalization (WGia W5, PfeV W2, oUrE W1)

Same-instance evaluation is the SkillsBench protocol. Beyond it, three settings test reuse where a recorded answer cannot carry over: 300 generated variants with resampled contents (Table 2), ALFWorld task types the teacher never solved (App. M), and a new τ-bench retail experiment where the student is evaluated only on held-out tasks disjoint from the source tasks. On τ-bench retail, PDI skills reach 80.0%, vs. 71.4% for skills without PDI and 68.6% with no skill.

We are grateful to all three reviewers for their time and constructive engagement, and to the AC for overseeing this discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.