- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper separates skill construction, which finds reusable action segments, from action promotion, which decides whether a given frozen policy may launch such a segment with a single decision. It argues that recurrence and executability describe the demonstrator and the environment rather than the deploying policy. In support, it shows that ALFWorld catalogues selected by these criteria realise their declared effect over 90% of the time yet reduce success from 0.861 to as low as 0.493. The proposed policy-relative admission (PRA) has three parts. First, it lifts frequent segments into typed operator sequences that a deterministic binder executes blindly, without the admissible-action oracle. Second, it certifies each candidate on held-out games by replaying the policy's history exactly and branching with and without the candidate. Third, it admits a candidate only if bootstrap lower bounds show effect fidelity , a positive decision saving , and success non-inferiority . Thresholds are frozen before held-out evaluation. On ALFWorld and WebShop, the evaluation uses a behaviour-cloned Qwen2.5-1.5B policy and the released SkillRL-7B agent. PRA is the only arm that saves decisions at non-inferior success in all four panels: 1.60 and 0.67 decisions per episode on ALFWorld, and on WebShop a 54% decision reduction for the cloned policy, whose score rises from 0.248 to 0.395. Ablations and four frozen policies show that the admitted sets depend on the policy.
Critical strengths
S1. A well-posed question with convincing evidence that executability is not value. Framing promotion as a constrained decision for one frozen policy (Eq. 2) is clean. The evidence that construction statistics are insufficient is persuasive. Without the value certificate, admitted abstractions realise their effect on 94.5% of commits, yet success falls to 0.493 (Table 2). All four effect/saving quadrants are populated (Table 4). The policy dependence of admission is also shown clearly: the drawer candidate is admitted for one Qwen policy and rejected for the other; swapping catalogues between policies costs either saving or success (Table 8); and SkillRL's released skill bank harms a frozen policy it was not trained with (Table 9).
S2. Rigorous and transparent methodology. Admission thresholds, seeds, and endpoints are fixed before the held-out splits are opened. Certification pairs branches on exactly replayed, hash-verified histories and uses one-sided bootstrap lower bounds. Development and held-out results are kept separate. The evaluation spans four policies that vary in capacity, family, and training. The full certification cost is reported, including an excluded experiment, and code is released.
Critical weaknesses
W1. Limited methodological novelty, and the headline claim largely holds by construction. At its core, PRA runs a standard paired non-inferiority test for each candidate against the deploying policy. Selecting macros by their measured utility for a specific planner on training problems has precedent in classical planning, for example the macro filtering in Macro-FF (Botea et al., 2005), which the paper cites only for its representation, and Wizard (Newton et al., 2007). These are not discussed. The claim that PRA "is the only method that cuts deployment-time decisions at non-inferior success" (l. 029–030) is unsurprising: every baseline (Handcrafted, SaS BPE port, uncertified pools) is untested against the policy, while PRA explicitly tests success non-inferiority. The missing baseline is a simpler policy-in-the-loop rule, such as greedy selection by point estimates. Moreover, the ablation in Table 2 suggests that the value certificate does nearly all the work. Without the effect certificate, success stays within the margin (−0.015, LCB −0.032) and the saving is slightly higher (1.82 vs. 1.60). Under the paper's own objective (Eq. 2), the effect certificate therefore contributes little, which weakens the claim that "both must hold."
W2. The practical significance is weak relative to the cost. On ALFWorld, PRA saves 1.60 of about 21 decisions per episode (7.5%) for the cloned policy. For the competent SkillRL agent it saves only 0.67 (3.1%, LCB 0.19). The larger WebShop gain comes mainly from forcing the cloned policy to buy once it reaches a purchasable page (App. M), which corrects a pathology of that particular policy rather than compressing recurring behaviour; for SkillRL, the saving is 1.34 decisions. Certification costs 67,470 policy decisions per seed, plus 72,999 for an excluded gating experiment, and break-even requires 42,247 episodes. App. J concedes that no ordering of these accounts "brings break-even within two orders of magnitude of either benchmark," whereas the main text says the charge "suits repeated deployment" (l. 407–408). Because certification is tied to one frozen policy, every policy update requires paying this cost again.
W3. The guarantees and the scope are narrow. The certificates are per-candidate. Each abstraction is certified as the only admitted one, and the success margin and selection level ( over 40 candidates, without multiplicity correction in the main arm) apply to each candidate separately. Deployment, however, uses the whole catalogue under a longest-match dispatcher, so interactions among abstractions and cumulative success loss are only checked empirically. With the family-wise correction, the saving drops to 0.81 decisions. Certification also requires exactly replayable, hash-verifiable environments, which real web and API settings rarely provide. Despite the "tool-agent" framing, the evidence covers only two text environments, policies of at most 7B parameters, and no actual tool or API use.
Questions for authors
- How does PRA differ in substance from planner-relative macro selection in classical planning, such as Macro-FF's filtering on training problems or Wizard's planner-based fitness?
- How does PRA compare with a simpler policy-in-the-loop baseline that greedily selects candidates by point estimates of saving and success? Under Eq. 2, is the variant without the effect certificate preferable?
- Can you provide a catalogue-level success guarantee, for example by jointly certifying the admitted set, and report how much success is lost when abstractions interact?
- How would certification work in environments that cannot be replayed exactly?
4 comments
Response to Reviewer kkbv (2/2)
Where the WebShop gain comes from (W2)
The buy entry corrects a weakness of the cloned policy. Its sessions account for 80% of the submitted selector's savings and all its score gain by attribution, so this includes improved task behavior as well as compression. The policy-specific result is that SkillRL-7B rejects the same entry and obtains savings from other abstractions.
Cost of obtaining that benefit.
Including the joint check, ALFWorld break-even is 46,337 episodes for the cloned policy and 81,556 for SkillRL-7B. Using saving lower bounds gives 66,690 and 373,180. The cloned WebShop estimate is about 2,100 sessions, reconstructed from archived pairs. For WebShop SkillRL, two seeds deploy Flat and the aggregate saving lower bound is zero. Accordingly, that bound establishes no finite break-even. Section 5.2 and Appendix L make the stable, repeated-deployment requirement explicit, including recertification after policy changes.
Testing the frozen catalogue with its dispatcher (W3 / Q3)
Check performed. After selection we freeze the catalogue and dispatcher, then compare that combination with Flat on 256 unused tasks. The nominal check requires a success lower bound of at least −0.05 and a positive saving lower bound. A failing seed returns to Flat, with no pruning and re-testing on the same block.
Observed result. This check catches a submitted-rule WebShop catalogue with 3 gains and 16 losses. On ALFWorld, an extra admission increases losses from 1 to 6, although the nominal check still passes. The latter is an effect in the combined catalogue; the entry was not also tested alone, so an interaction term cannot be isolated.
We report exact-success-bound sensitivity alongside the nominal procedure. For revised PRA, nominal/exact passes are 3/3 versus 1/3 on ALFWorld cloned, 3/3 versus 2/3 on ALFWorld SkillRL, and 3/3 versus 1/3 on WebShop cloned. The sole non-empty WebShop SkillRL catalogue passes both. Table 5 includes the lower savings after the stricter check.
Scope of the result. The main check is approximate. The exact bound concerns binary success of the frozen catalogue under the sampling assumptions, rather than fidelity, bootstrap savings or all deployment distributions. ALFWorld's joint block uses unused seen-scene training tasks; valid-unseen is evaluated separately. Appendix D reports the screening bounds' coverage limitations, including correlated commits and failure-dependent counts.
Environments without exact replay (Q4)
Exact replay enables the paired estimator used here. Snapshots or randomized independent episodes could support different estimators, but the completed evidence is for two replayable text environments and policies up to 7B. We have made that implementation scope explicit.
The new controls therefore sharpen the deployment decision: whether this policy should use this catalogue, given requirements on task success, execution semantics and amortized cost.
Follow-up on the contribution and added comparisons
Thank you again for your time in reviewing our work. We hope our response has clarified the contribution relative to prior macro selection. We would like to connect that positioning to the added evidence.
- First, the revision makes PRA's contribution relative to Macro-FF and Wizard explicit. We separate reliable execution from usefulness to a frozen language policy and test both before admitting an action. The same faithfully executed buy action improves the cloned policy's certification success but reduces SkillRL-7B's, so it is admitted for the former and rejected for the latter.
- To test what the selection conditions contribute, we compared point-estimate selectors using the same candidates and evaluation budget. On WebShop, PRA achieves 94.9% fidelity versus 76.3% for the effect-free mean selector, while saving 14.72 versus 15.47 decisions. This makes the effect requirement concrete: it checks whether the declared action is reliably realized, alongside the separate checks on decision savings and task success.
- Finally, we added a frozen-catalogue check to assess joint deployment. This tests the selected combination, but does not perform joint greedy search; the new selectors remain per-candidate controls.
We would be glad to discuss any remaining concern about this distinction or the comparisons.
Thank you for the rebuttal, I would like to maintain the overall accessment.
Response to Reviewer kkbv (1/2)
Relation to Macro-FF and Wizard (W1 / Q1)
Thank you for pointing out the planner-relative selection in Macro-FF and Wizard. We agree that these are direct precedents. Macro-FF measures macro utility through search effort on training problems, while Wizard learns macros and macro sets for a planner–domain pair. Section 2 now explains this relationship. Our contribution builds on that idea by asking what should be admitted for a frozen language policy, as illustrated below.
For a frozen language policy, a faithfully executed sequence can still cause task loss. PRA measures whether a blind sequence realizes its declared effect and whether using it helps that policy. The new joint check evaluates the catalogue with its dispatcher. The option-then-buy entry raises the cloned policy's certification success by 0.086–0.098 across seeds, but lowers SkillRL-7B's by 0.061–0.156. It is admitted for the former and rejected for the latter despite faithful execution under both.
Testing simpler selection (Q2)
Comparison performed. We tested point-estimate selectors on the same candidates, records and selection budget. PRA-Mean uses means instead of the submitted bounds. The effect-free version additionally drops the effect condition. Each is evaluated directly and after the same joint check. Held-out means after the nominal check are:
What the comparison establishes. The mean rule saves more on ALFWorld. Under the original decision-count objective alone, an effect-free selector can be preferable. On WebShop it saves 15.47 versus 14.72 decisions, but 265/1,118 commits abort versus 35/681 for PRA. An abort returns control after commands have executed, without rollback. The effect requirement therefore expresses a semantic condition on the action offered to the policy. We now state this condition alongside Eq. 2. PRA treats faithful execution and utility for the target policy as separate admission requirements, so an action's compression benefit alone does not establish its suitability for deployment.
These policy-tested controls also address the concern that the comparison favors PRA by construction. Requiring non-inferiority during selection does not make non-inferiority on separate held-out tasks true by construction. The added comparisons test what simpler policy-relative rules achieve and show why decision savings alone do not establish reliable execution of the declared action. They are per-candidate mean selectors, rather than a joint greedy search over catalogue additions. Figure 2 and Table 5 report the complete comparisons, including uncertainty and Flat returns. Joint greedy selection remains a distinct comparison.