- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SKILL.nb stores workflows in versioned notebooks. Each notebook combines natural-language procedures and executable steps with validation gates and execution records. A lifecycle policy controls workflow release and retirement. It also decides when to formalize a step or demote it to natural-language execution. At runtime, failed code can give way to a procedure or the step's intent, subject to the execution rules. The paper evaluates online task success, reuse and repair, and transfer of stored workflows across GitLab versions.
Critical strengths
-
Repeated use is central to the evaluation. A workflow may solve its first task yet become unreliable after repair or reuse; retention, recovery, and regression measurements expose that failure. This is a more informative test of a persistent workflow library than initial success alone.
-
The GitLab version changes provide a concrete test of stale workflows. On version 18.9, SKILL.nb solves 110/180 tasks fresh and 111/180 with reuse. Fresh runs help control for target-version difficulty. Restoring the source snapshot before each reuse task also rules out accumulated learning on that version as the explanation.
-
The ablations test substantive design choices: natural-language-only and code-only execution, removal of gates or cleanup, and fixed versus calibrated thresholds. These comparisons support a contribution from execution and maintenance policies, beyond changing the storage format.
-
The incident audit makes gate failures visible, including false positives that a final success metric would obscure. The paper also reports the higher aggregate inference cost and acknowledges the limits of replay calibration. These disclosures help delimit the practical claim.
Critical weaknesses
-
The main comparisons mix several sources of variation. Inference settings and update paths differ across methods, and reuse and repair are measured on method-specific task sets. This is a comparison of complete systems under their native configurations. The effect of the lifecycle policy remains unclear. A matched-compute comparison on shared repair cases is still missing.
-
The initial gain is 53.7% versus 49.8%, with a reported McNemar p-value of 0.029. However, persistent updates make tasks dependent, and the stated tests do not model that dependence. Only one task-order stream is reported. Independent order repeats are needed to establish whether this 3.9-point gain survives changes in the learning history.
-
Threshold calibration excludes the future runtime cost of suppressing an action. The violation-rate denominator includes every logged opportunity, even cases the policy suppresses. This can favor policies that admit fewer updates. The paper needs admission counts, conditional violation rates, and deferral frequencies to show what the selected thresholds actually do.
-
The supplement is a simplified implementation, as its README states. The code uses fixed success/failure thresholds and includes lightweight evaluation examples. The full group-calibrated controller and benchmark pipeline are absent. The supplied artifact is insufficient to verify the policy or reproduce the main results.
Questions for authors
-
Do the advantages persist when reuse and repair use a common task cohort and inference settings or token budgets are matched? Please report results across independent online task orders.
-
For each lifecycle decision, how many cases are usable and how many are admitted? Please report the selected thresholds, deferral frequency, and violation rate among admitted actions. How does the objective avoid suppressing useful updates when their future runtime benefit is absent from the cost estimate?
-
Can you supply the actual calibration and evaluation code, exact prompts, configurations, and per-task results? Please distinguish this code from the simplified supplement. How does the 30-step budget count a code cell with several browser actions?
3 comments
W1 / Q1: Reuse and repair on identical tasks, with matched settings
Scored on the same tasks, SKILL.nb succeeds more often on both reuse and repair, and spends fewer tokens per success on repair.
| Task set (from Table 1) | Order | SKILL.nb | ReasoningBank | Difference | Solved by one only (SKILL.nb / RB) | McNemar p | Tokens per success (SKILL.nb / RB) |
|---|---|---|---|---|---|---|---|
| Reuse: 164 both solved | 2 | 92.7 | 84.8 | +7.9 | 14 / 1 | 0.002 | 59k / 51k |
| 3 | 91.5 | 86.0 | +5.5 | 14 / 5 | 0.067 | 65k / 49k | |
| Repair: 136 both failed | 2 | 64.7 | 50.0 | +14.7 | 32 / 12 | 0.004 | 110k / 180k |
| 3 | 64.0 | 51.5 | +12.5 | 32 / 15 | 0.020 | 120k / 172k |
- Identical tasks: both methods are scored on the same tasks, complementing the method-specific denominators of Figure 1(b, c). The repair difference is larger than the reuse difference, and the reuse difference in Order 3 is not significant (p = 0.067).
- Compute: inference settings are matched; token use is reported, not capped. On the shared repair tasks, SKILL.nb uses fewer tokens per task (71–77k vs 88–90k) and still succeeds more often, so its repair lead does not come from extra compute. On reuse it spends more per success (59–65k vs 49–51k). Tokens include SKILL.nb's maintenance and ReasoningBank's judge and memory extraction, and failed attempts count.
- Update paths and the lifecycle policy: each method keeps its native update path, since that path is the method being compared. The policy's own effect is isolated within SKILL.nb by the ablations that disable lifecycle cleanup (Table 5) or replace the calibrated thresholds with fixed ones (App. C.2).
- Setup: in Table 1, both methods solve 372 tasks and both fail 312; we drew 300 of these 684, stratified by outcome and site. Each method reruns them once per order (Orders 2 and 3, see W2), both times from its own end-of-round state, with updates on and sites reset before every task (App. C.8).
W2 / Q1: Independent task orders
SKILL.nb leads ReasoningBank in every order. We ran both methods on two further random task orders, each starting without persistent state and applied to both methods (App. C.7, Table 2).
| Order | ReasoningBank | SKILL.nb | Difference | Solved by SKILL.nb only / RB only | McNemar p |
|---|---|---|---|---|---|
| 1 (Table 1) | 51.1 | 56.3 | +5.2 | 85 / 43 | 2.9×10⁻⁴ |
| 2 | 47.0 | 55.5 | +8.5 | 124 / 55 | 3.7×10⁻⁷ |
| 3 | 49.9 | 56.4 | +6.5 | 101 / 48 | 2.0×10⁻⁵ |
- Dependence: we agree that a test within one order does not model the dependence that persistent updates create. The orders are the independent repeats: the learning history differs in each, and the lead stays between 5.2 and 8.5 points (6.7 on average).
- Stability: SKILL.nb varies by 0.9 points across orders (55.5–56.4%), ReasoningBank by 4.1 (47.0–51.1%).
W3 / Q2: What the thresholds actually do, and suppressed updates
We agree with the concern: the denominator n includes suppressed cases, and the maintenance-cost objective also favors admitting fewer updates. To show what the procedure does in practice, we replayed it (§3.3, App. A.7) over every lifecycle decision our controller logged on the 106 Reddit tasks over three task orders; k counts admitted cases that later failed (App. C.6). The filter removes loose thresholds but leaves a wide feasible range, and within it, cost ranking favors admitting fewer updates, as the reviewer anticipates.
- Selected thresholds: each admission adds maintenance cost, so cost ranking selects the most restrictive feasible threshold (τ = 22, pooled and in both feasible groups), which formalizes 3 of 188 steps. The least restrictive feasible thresholds differ by group: create steps need 7 successful uses, edit steps 3; the census table below reports both ends.
- Violation rate among admitted (k/n_adm): 0–6.5% at the least restrictive feasible thresholds, which admit 24–31% of steps. Formalizing after 2 successful uses would admit 128 of 188 steps, 14 of which later fail (10.9% of admitted, 7.4% of all usable cases); the Wilson upper bound on the latter, 11.2%, exceeds 5%, so the filter rejects it.
- Deferral frequency: 11% of formalization cases (the 20 read-group cases, too few for the 5% bound), all 97 release decisions (16.5% of released workflows were later used by a failed task) and all demotion and retirement decisions (no accepted-repair calibration cases). Deferral routes a decision to maintenance review, so no case is silently dropped.
- Two rates, kept apart: the filter bounds k/n; the census table reports k/n_adm separately. In App. A.7's example, 2 violations among 20 admitted of 100 usable cases give k/n = 2% but k/n_adm = 10%, and that candidate is rejected (Wilson upper bound ≈ 5.86%).
- Suppressed useful updates: the objective itself does not prevent this; two features limit the loss. First, a threshold decides how earlier experience is used, not whether the agent sees it: every execution reads the per-step notes of earlier executions, including those whose proposals are pending or were not released (App. D.1). Release changes how a workflow is used: a released workflow is followed under its gates, a formalized step runs as code, and deviating from it requires a repair proposal; unreleased notes are hints the agent can depart from freely. A suppressed update therefore costs binding, gated reuse and its lower token use (SKILL.nb's tokens per success fall from 52k to 36k over five rounds as workflows are released; App. B.3, gpt-5.3-codex), not the knowledge itself. Second, crossing a threshold triggers review, not a write. In a separate run (App. C.2: hard subset, gpt-5.3-codex), the calibrated policy is not the cheapest: strict fixed thresholds have the lowest round-3 maintenance-token measure (47k vs 67k) but about 31% success vs 38.4%.
Decision census at V_max = 0.05 (admitted at the least and most restrictive feasible thresholds; k/n_adm at the least restrictive)
| Decision | Group | Usable n | Feasible τ | Admitted (least / most restrictive) | k/n_adm | Deferred |
|---|---|---|---|---|---|---|
| Formalize step | pooled | 188 | 5–22 | 46 / 3 | 6.5% | – |
| create | 94 | 7–22 | 29 / 3 | 3.4% | – | |
| edit | 74 | 3–22 | 21 / 0 | 0.0% | – | |
| read | 20 | none | – | – | 20 | |
| Release workflow | pooled | 97 | none | – | – | 97 |
| Demote step | pooled | 13 | none | – | – | 13 |
| Retire workflow | pooled | 121 | none | – | – | 121 |
Q3: The 30-step budget
A code cell does not count as one step: every browser action inside it that changes the page counts separately. A cell that fills a field and clicks Save uses 2 of the 30 steps, exactly as if the agent had issued the two actions itself, so running a workflow as code gives SKILL.nb no extra browser actions within the budget. We now define this in App. B.1.
- Counted: click, fill, select option, key press, and navigation (go to URL, back, forward).
- Not counted: hover, scroll, page reads and snapshots, and gate checks, which observe the page without changing it.
- Same rule for every method: all methods act through the same Playwright browser, where the harness counts steps identically (App. B.1). This is at least as strict as the baselines' native setups: in BrowserGym, one step is one environment call and can carry several actions.
- In practice, a cell that issued at least one counted action used a median of 2 steps (90th percentile 4; per-cell counts in the released results).
We thank the reviewer for the careful reading of repeated use, version drift and the ablations. We address each point with new evidence and revisions to the paper. In short:
New revision (uploaded): abstract, introduction and §4 (new online-round numbers), Table 1 (new runs), Table 2 and App. C.7 (independent orders), Table 4 and App. C.8 (identical tasks), App. B.1 (model per experiment; step budget), App. C.6 (census), App. D (prompts).
Details follow in the next comments (W1 and W2, then W3/Q2 and Q3); the revised PDF gives the full setup.
New runs for W1 and W2. Because gpt-5.3-codex is no longer available, we reran the full 812-task online round for all four methods with gpt-6.1-terra, using the same settings for every method (medium reasoning effort, low verbosity, same 30-step budget and harness). Table 1 now reports these runs (56.3% vs 51.1% for ReasoningBank); the 53.7% vs 49.8% round cited in the review remains as round 1 of the gpt-5.3-codex lifecycle runs (Fig. 1a). Tables 2 and 4 below use the new model and settings; the ablations cited for the lifecycle policy (Table 5, App. C.2) are the earlier gpt-5.3-codex runs.