acceptodds
Under review as a conference paper at ICLR 2027

PRA: Certifying Skills Before They Become Tool-Agent Actions

Abstract

Tool agents repeat short action sequences, and every step of such a sequence costs a policy call. Recurring behaviour, which we call a skill, is therefore an opportunity to reduce calls. A large body of work asks what is worth constructing from demonstrations. However, whether a deployed policy should then launch that skill with one decision is a different question. Existing criteria cannot answer it, because recurrence and executability describe the demonstrator and the environment, not the policy that will pay. On ALFWorld, catalogues selected for recurrence or executability realise their declared effect more than 90% of the time yet cut task success from 0.861 to as low as 0.493. To address this, we propose policy-relative admission (PRA). PRA separates skill construction from action promotion and settles promotion as a decision about one frozen policy. Candidates are lifted into operator sequences that a deterministic binder executes blind, so runnability is measured rather than assumed. Certification then asks two questions on held-out games. One asks whether committed execution realises the declared effect, and the other whether exposing the action lowers decisions without losing task success beyond a non-inferiority margin. Experimental results show that the same behaviour is an action for one policy and not for another, since the admitted set moves with policy capacity and family and with how the policy was trained. Both certificates reject distinct failure modes, and neither subsumes the other. Under a behaviour-cloned policy and a released RL-trained agent alike, PRA is the only method that cuts deployment-time decisions at non-inferior success in both environments. On WebShop it removes 54% of the cloned policy’s decisions and raises its official score from 0.248 to 0.395. Our work turns promotion from a property assumed at construction time into a decision a deployment can audit.

Primary area:
reinforcement learning
Keywords:
action abstraction, skill certification, language agents, hierarchical decision-making, non-inferiority testing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_kkbvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper separates skill construction, which finds reusable action segments, from action promotion, which decides whether a given frozen policy may launch such a segment with a single decision. It argues that recurrence and executability describe the demonstrator and the environment rather than the deploying policy. In support, it shows that ALFWorld catalogues selected by these criteria realise their declared effect over 90% of the time yet reduce success from 0.861 to as low as 0.493. The proposed policy-relative admission (PRA) has three parts. First, it lifts frequent segments into typed operator sequences that a deterministic binder executes blindly, without the admissible-action oracle. Second, it certifies each candidate on held-out games by replaying the policy's history exactly and branching with and without the candidate. Third, it admits a candidate only if bootstrap lower bounds show effect fidelity F(z)≥0.75F(z)\ge 0.75, a positive decision saving VD(z)>0V_D(z)>0, and success non-inferiority VS(z)≥−0.05V_S(z)\ge -0.05. Thresholds are frozen before held-out evaluation. On ALFWorld and WebShop, the evaluation uses a behaviour-cloned Qwen2.5-1.5B policy and the released SkillRL-7B agent. PRA is the only arm that saves decisions at non-inferior success in all four panels: 1.60 and 0.67 decisions per episode on ALFWorld, and on WebShop a 54% decision reduction for the cloned policy, whose score rises from 0.248 to 0.395. Ablations and four frozen policies show that the admitted sets depend on the policy.

Critical strengths

S1. A well-posed question with convincing evidence that executability is not value. Framing promotion as a constrained decision for one frozen policy (Eq. 2) is clean. The evidence that construction statistics are insufficient is persuasive. Without the value certificate, admitted abstractions realise their effect on 94.5% of commits, yet success falls to 0.493 (Table 2). All four effect/saving quadrants are populated (Table 4). The policy dependence of admission is also shown clearly: the drawer candidate is admitted for one Qwen policy and rejected for the other; swapping catalogues between policies costs either saving or success (Table 8); and SkillRL's released skill bank harms a frozen policy it was not trained with (Table 9).

S2. Rigorous and transparent methodology. Admission thresholds, seeds, and endpoints are fixed before the held-out splits are opened. Certification pairs branches on exactly replayed, hash-verified histories and uses one-sided bootstrap lower bounds. Development and held-out results are kept separate. The evaluation spans four policies that vary in capacity, family, and training. The full certification cost is reported, including an excluded experiment, and code is released.

Critical weaknesses

W1. Limited methodological novelty, and the headline claim largely holds by construction. At its core, PRA runs a standard paired non-inferiority test for each candidate against the deploying policy. Selecting macros by their measured utility for a specific planner on training problems has precedent in classical planning, for example the macro filtering in Macro-FF (Botea et al., 2005), which the paper cites only for its representation, and Wizard (Newton et al., 2007). These are not discussed. The claim that PRA "is the only method that cuts deployment-time decisions at non-inferior success" (l. 029–030) is unsurprising: every baseline (Handcrafted, SaS BPE port, uncertified pools) is untested against the policy, while PRA explicitly tests success non-inferiority. The missing baseline is a simpler policy-in-the-loop rule, such as greedy selection by point estimates. Moreover, the ablation in Table 2 suggests that the value certificate does nearly all the work. Without the effect certificate, success stays within the margin (−0.015, LCB −0.032) and the saving is slightly higher (1.82 vs. 1.60). Under the paper's own objective (Eq. 2), the effect certificate therefore contributes little, which weakens the claim that "both must hold."

W2. The practical significance is weak relative to the cost. On ALFWorld, PRA saves 1.60 of about 21 decisions per episode (7.5%) for the cloned policy. For the competent SkillRL agent it saves only 0.67 (3.1%, LCB 0.19). The larger WebShop gain comes mainly from forcing the cloned policy to buy once it reaches a purchasable page (App. M), which corrects a pathology of that particular policy rather than compressing recurring behaviour; for SkillRL, the saving is 1.34 decisions. Certification costs 67,470 policy decisions per seed, plus 72,999 for an excluded gating experiment, and break-even requires 42,247 episodes. App. J concedes that no ordering of these accounts "brings break-even within two orders of magnitude of either benchmark," whereas the main text says the charge "suits repeated deployment" (l. 407–408). Because certification is tied to one frozen policy, every policy update requires paying this cost again.

W3. The guarantees and the scope are narrow. The certificates are per-candidate. Each abstraction is certified as the only admitted one, and the success margin and selection level (αsel=0.05\alpha_{\text{sel}}=0.05 over 40 candidates, without multiplicity correction in the main arm) apply to each candidate separately. Deployment, however, uses the whole catalogue under a longest-match dispatcher, so interactions among abstractions and cumulative success loss are only checked empirically. With the family-wise correction, the saving drops to 0.81 decisions. Certification also requires exactly replayable, hash-verifiable environments, which real web and API settings rarely provide. Despite the "tool-agent" framing, the evidence covers only two text environments, policies of at most 7B parameters, and no actual tool or API use.

Questions for authors
  1. How does PRA differ in substance from planner-relative macro selection in classical planning, such as Macro-FF's filtering on training problems or Wizard's planner-based fitness?
  2. How does PRA compare with a simpler policy-in-the-loop baseline that greedily selects candidates by point estimates of saving and success? Under Eq. 2, is the variant without the effect certificate preferable?
  3. Can you provide a catalogue-level success guarantee, for example by jointly certifying the admitted set, and report how much success is lost when abstractions interact?
  4. How would certification work in environments that cannot be replayed exactly?
4 comments
AuthorsOpenReview ↗

Response to Reviewer kkbv (1/2)

Relation to Macro-FF and Wizard (W1 / Q1)

Thank you for pointing out the planner-relative selection in Macro-FF and Wizard. We agree that these are direct precedents. Macro-FF measures macro utility through search effort on training problems, while Wizard learns macros and macro sets for a planner–domain pair. Section 2 now explains this relationship. Our contribution builds on that idea by asking what should be admitted for a frozen language policy, as illustrated below.

For a frozen language policy, a faithfully executed sequence can still cause task loss. PRA measures whether a blind sequence realizes its declared effect and whether using it helps that policy. The new joint check evaluates the catalogue with its dispatcher. The option-then-buy entry raises the cloned policy's certification success by 0.086–0.098 across seeds, but lowers SkillRL-7B's by 0.061–0.156. It is admitted for the former and rejected for the latter despite faithful execution under both.

Testing simpler selection (Q2)

Comparison performed. We tested point-estimate selectors on the same candidates, records and selection budget. PRA-Mean uses means instead of the submitted bounds. The effect-free version additionally drops the effect condition. Each is evaluated directly and after the same joint check. Held-out means after the nominal check are:

Panel / selectorDecisions savedFidelitySuccess difference
ALFWorld: PRA1.680.886+0.000
ALFWorld: PRA-Mean2.440.852+0.002
ALFWorld: Mean w/o effect1.770.808−0.007
WebShop: PRA14.720.949+0.040
WebShop: PRA-Mean10.160.950+0.013
WebShop: Mean w/o effect15.470.763+0.027

What the comparison establishes. The mean rule saves more on ALFWorld. Under the original decision-count objective alone, an effect-free selector can be preferable. On WebShop it saves 15.47 versus 14.72 decisions, but 265/1,118 commits abort versus 35/681 for PRA. An abort returns control after commands have executed, without rollback. The effect requirement therefore expresses a semantic condition on the action offered to the policy. We now state this condition alongside Eq. 2. PRA treats faithful execution and utility for the target policy as separate admission requirements, so an action's compression benefit alone does not establish its suitability for deployment.

These policy-tested controls also address the concern that the comparison favors PRA by construction. Requiring non-inferiority during selection does not make non-inferiority on separate held-out tasks true by construction. The added comparisons test what simpler policy-relative rules achieve and show why decision savings alone do not establish reliable execution of the declared action. They are per-candidate mean selectors, rather than a joint greedy search over catalogue additions. Figure 2 and Table 5 report the complete comparisons, including uncertainty and Flat returns. Joint greedy selection remains a distinct comparison.

AuthorsOpenReview ↗

Response to Reviewer kkbv (2/2)

Where the WebShop gain comes from (W2)

The buy entry corrects a weakness of the cloned policy. Its sessions account for 80% of the submitted selector's savings and all its score gain by attribution, so this includes improved task behavior as well as compression. The policy-specific result is that SkillRL-7B rejects the same entry and obtains savings from other abstractions.

Cost of obtaining that benefit.

Including the joint check, ALFWorld break-even is 46,337 episodes for the cloned policy and 81,556 for SkillRL-7B. Using saving lower bounds gives 66,690 and 373,180. The cloned WebShop estimate is about 2,100 sessions, reconstructed from archived pairs. For WebShop SkillRL, two seeds deploy Flat and the aggregate saving lower bound is zero. Accordingly, that bound establishes no finite break-even. Section 5.2 and Appendix L make the stable, repeated-deployment requirement explicit, including recertification after policy changes.

Testing the frozen catalogue with its dispatcher (W3 / Q3)

Check performed. After selection we freeze the catalogue and dispatcher, then compare that combination with Flat on 256 unused tasks. The nominal check requires a success lower bound of at least −0.05 and a positive saving lower bound. A failing seed returns to Flat, with no pruning and re-testing on the same block.

Observed result. This check catches a submitted-rule WebShop catalogue with 3 gains and 16 losses. On ALFWorld, an extra admission increases losses from 1 to 6, although the nominal check still passes. The latter is an effect in the combined catalogue; the entry was not also tested alone, so an interaction term cannot be isolated.

We report exact-success-bound sensitivity alongside the nominal procedure. For revised PRA, nominal/exact passes are 3/3 versus 1/3 on ALFWorld cloned, 3/3 versus 2/3 on ALFWorld SkillRL, and 3/3 versus 1/3 on WebShop cloned. The sole non-empty WebShop SkillRL catalogue passes both. Table 5 includes the lower savings after the stricter check.

Scope of the result. The main check is approximate. The exact bound concerns binary success of the frozen catalogue under the sampling assumptions, rather than fidelity, bootstrap savings or all deployment distributions. ALFWorld's joint block uses unused seen-scene training tasks; valid-unseen is evaluated separately. Appendix D reports the screening bounds' coverage limitations, including correlated commits and failure-dependent counts.

Environments without exact replay (Q4)

Exact replay enables the paired estimator used here. Snapshots or randomized independent episodes could support different estimators, but the completed evidence is for two replayable text environments and policies up to 7B. We have made that implementation scope explicit.

The new controls therefore sharpen the deployment decision: whether this policy should use this catalogue, given requirements on task success, execution semantics and amortized cost.

AuthorsOpenReview ↗

Follow-up on the contribution and added comparisons

Thank you again for your time in reviewing our work. We hope our response has clarified the contribution relative to prior macro selection. We would like to connect that positioning to the added evidence.

  • First, the revision makes PRA's contribution relative to Macro-FF and Wizard explicit. We separate reliable execution from usefulness to a frozen language policy and test both before admitting an action. The same faithfully executed buy action improves the cloned policy's certification success but reduces SkillRL-7B's, so it is admitted for the former and rejected for the latter.
  • To test what the selection conditions contribute, we compared point-estimate selectors using the same candidates and evaluation budget. On WebShop, PRA achieves 94.9% fidelity versus 76.3% for the effect-free mean selector, while saving 14.72 versus 15.47 decisions. This makes the effect requirement concrete: it checks whether the declared action is reliably realized, alongside the separate checks on decision savings and task success.
  • Finally, we added a frozen-catalogue check to assess joint deployment. This tests the selected combination, but does not perform joint greedy search; the new selectors remain per-candidate controls.

We would be glad to discuss any remaining concern about this distinction or the comparisons.

Thank you for the rebuttal, I would like to maintain the overall accessment.

Reviewer_1bKvOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes policy-relative admission, a procedure for deciding whether a reusable action sequence should become an automatically executed macro-action for a frozen tool-agent policy. PRA evaluates candidates using two criteria: whether execution reliably produces the declared effect, and whether enabling the candidate reduces policy decisions while keeping task success within a predefined non-inferiority margin. Experiments on ALFWorld and WebShop demonstrate that frequently occurring or reliably executable sequences can nevertheless harm task performance, whereas PRA-selected catalogues reduce decisions while satisfying the evaluated success constraint. The central contribution is the distinction between constructing reusable skills and determining their value for a particular deployed policy.

Critical strengths
  1. The paper identifies a clear and useful distinction between skill construction and deployment value. A sequence can be common in successful trajectories and executable in the environment without being beneficial whenever a particular policy triggers it. Framing skill promotion as a decision about a frozen policy gives the paper a coherent research question and a practical contribution: deciding which existing routines to enable without retraining the agent.
  2. The main finding holds across multiple policies and both environments. All four main experimental panels report positive lower bounds on decision savings while remaining within the predefined success margin.
Critical weaknesses
  1. Certifying candidates individually does not certify their combined catalogue. Equation 2 states an objective and success constraint for the complete catalogue. However, certification evaluates each candidate separately against the primitive policy, and deployment then enables all admitted candidates together using longest-match dispatch. Earlier macros can change the states where later macros execute, and overlapping macros can compete. Even without complicated interactions, two candidates could each cause a four-percentage-point success loss on different tasks and satisfy the five-point margin individually, while their combination loses eight points. The separate held-out evaluation provides evidence that the reported catalogues work, but the admission rule itself does not establish the catalogue-level constraint. A final joint admission check would make the certification claim more complete.
  2. The upfront cost is substantial relative to the ALFWorld savings. Proposal and certification require 67,470 policy decisions per seed, while the held-out saving is 1.60 decisions per episode. The authors consequently report a break-even point of approximately 42,247 deployment episodes. This can be reasonable for a frequently reused, stable policy, but the economic benefit is less clear for smaller deployments or policies that change frequently and require recertification. The paper deserves credit for reporting this cost. Nevertheless, the amortization requirement materially limits practical applicability, and comparable cost analyses for the other policy–environment combinations would strengthen the deployment argument.
Questions for authors
  1. How reliable are the percentile bootstrap lower bounds when the number of firing games is small or all observed commits are faithful? Could the authors evaluate their coverage or consider alternative confidence bounds?

  2. In WebShop, some declared effects are relatively weak (e.g., click buy only requires episode termination). How meaningful is effect fidelity under these definitions, and would stronger semantic predicates change the results?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 1bKv (1/2)

Thank you for the two-candidate counterexample. We have implemented the final joint check you suggest: the frozen catalogue is evaluated together with the dispatcher that will deploy it. This makes the deployment decision depend on aggregate task outcomes, rather than the sum of candidate-wise allowances.

Checking the combined catalogue (W1)

What we implemented. The catalogue and dispatcher are compared with Flat on 256 unused tasks. The nominal rule requires a paired success lower bound of at least −0.05 and a positive saving lower bound. Otherwise that seed deploys Flat. The protocol was frozen before these rebuttal rollouts, and no catalogue is pruned and re-tested on the same block.

What changed in deployment. On WebShop seed 42, the submitted selector's catalogue yields 3 gains and 16 losses and is rejected. Revised PRA excludes its option→option entry and passes. On ALFWorld seed 42, adding one entry changes 0 gains/1 loss into 1 gain/6 losses. Since the extra entry was not also tested alone on this block, that comparison measures its effect in the catalogue without isolating an interaction term.

Because the nominal joint check uses an approximate success bound, we also evaluated a conservative exact-success-bound alternative. The comparison below shows how the deployment decision changes under the stricter success criterion:

Revised PRANominal passesExact-success passes
ALFWorld, cloned3/31/3
ALFWorld, SkillRL-7B3/32/3
WebShop, cloned3/31/3
WebShop, SkillRL-7B1/1 non-empty1/1 non-empty

The two empty WebShop SkillRL catalogues deploy Flat. The exact-success variant gives aggregate savings of 0.69 on ALFWorld cloned and 5.09 on WebShop cloned, including Flat returns. Table 5 also retains the submitted bootstrap selector, which passes the exact-success check on all three ALFWorld cloned seeds.

Scope of the bounds. The nominal success bound is Tango's approximate bound. The exact alternative subtracts an upper Clopper–Pearson loss bound from a lower gain bound, each at alpha/2. Its finite-sample property concerns binary success under the sampling assumptions. It does not make fidelity or bootstrap savings exact. ALFWorld uses unused seen-scene training tasks and WebShop unused training-range sessions. Seeds share tasks; valid-unseen performance is evaluated separately in Table 1.

The zero-failure boundary (Q1)

Turning to your zero-failure concern, we examined coverage for both the submitted bootstrap and the revised fidelity bound. At 31 firing games and true fidelity 0.98, the submitted bootstrap covers only 51% in simulation. The revised cluster-adjusted bound covers at least 0.959 in the approximately one-commit-per-game simulations, but coverage falls to 0.908 with correlated commits and about 0.5 with failure-dependent counts. We therefore describe it as a nominal screening criterion and report the full coverage analysis in Appendix D.

Success now uses Eq. 2's per-episode scale, where non-firing episodes contribute zero. This differs from the submitted firing-episode constraint, and both scales are documented. The separate joint check directly tests the catalogue-level success requirement raised in your review.

AuthorsOpenReview ↗

Response to Reviewer 1bKv (2/2)

When admission repays its cost (W2)

Appendix L now includes the joint check in admission cost and gives separate amortization figures:

PanelBreak-even at mean savingAt the saving's lower bound
ALFWorld, cloned46,337 episodes66,690
ALFWorld, SkillRL-7B81,556 episodes373,180
WebShop, clonedAbout 2,100 sessionsReconstructed estimate

WebShop costs are marked as estimates because decisions were reconstructed from stored pairs. For WebShop SkillRL, two seeds return to Flat and the aggregate saving lower bound is zero, so that bound gives no finite break-even. The same holds for ALFWorld cloned under exact-success-controlled deployment. The appropriate use case is a stable policy reused enough to amortize this cost, with recertification after policy changes.

What the WebShop effect checks (Q2)

The current buy effect checks execution semantics: buying terminates the session. Goal satisfaction is tested separately through task success. The policy contrast shows why both matter. The buy entry benefits the cloned policy, but is rejected for SkillRL-7B because its success effect is negative despite faithful execution.

The observed purchase outcomes. For the submitted selector's buy-entry sessions, purchase reward is 0.586 versus Flat's 0.242, and at least as high as Flat's in 86% of sessions. Flat never clicks buy within the horizon in about 66% of this group. These sessions comprise 46.5% of evaluation sessions and account for 80% of savings by attribution. This explains the cloned policy's benefit, while SkillRL's savings come from other entries.

A stronger predicate. A stronger predicate could use observable user requirements and page state, without hidden evaluator information. The present binder and stored records do not support evaluating that extension. Sections 5.2 and 5.5 now separate execution fidelity from task success and state the purchase attribution, so the termination predicate is not read as a complete goal verifier.

AuthorsOpenReview ↗

Follow-up on the joint check and deployment conditions

We appreciate the time you have devoted to reviewing our paper. We hope our response has made clear how we addressed your catalogue-level example. The key change is that the catalogue and dispatcher are now checked together before deployment.

  • To test the combined behavior directly, we compare the frozen catalogue and dispatcher against Flat on 256 unused tasks. A failed check returns the seed to Flat, without pruning and retesting on that block. This matters in practice: the check rejects a submitted-rule WebShop catalogue with 3 gains and 16 losses, even though its components passed individual admission.
  • Separately, we corrected the zero-failure bootstrap treatment and distinguished the levels of assurance. The nominal bounds remain approximate. The conservative exact alternative concerns binary task success, not fidelity, savings or unseen-scene generalization. Thus, the joint check addresses composition without implying a guarantee for every reported metric.
  • We also separate WebShop termination from goal achievement, which is assessed separately, and include the joint check in the deployment cost. The cost is amortized through repeated use of the same policy.

We would welcome your thoughts on whether this addresses the composition concern, and are happy to discuss the scope of the check further.

Reviewer_22t5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether a sequence of actions can safely become a reusable skill for a particular frozen agent. It tests each proposed skill against two requirements: 1) effect fidelity: does the skill reliably produce its declared outcome? 2) policy-relative value: does it reduce model decisions without materially reducing task success? Candidate skills are proposed from frequently repeated action sequences, tested on held-out games, and only those passing both checks are made available to the agent. On ALFWorld and WebShop, with a fine-tuned small model and an existing RL-trained agent, the certified skills save decisions without losing tasks, whereas skill sets chosen by frequency, executability or hand-design reduce success. The authors also show that which skills pass depends on the agent: the same skill can help one policy and hurt another. The savings are largest for the weaker policy and modest for the stronger one.

Critical strengths
  • The paper makes a useful empirical distinction between reliable skill execution and usefulness to the policy deploying it. In the ALFWorld ablation, selecting skills using effect fidelity alone yields high fidelity but substantially reduces task success. This provides a concrete motivation for assessing downstream policy performance when selecting skills for a catalogue.
  • Several experimental choices strengthen the evidence: admission thresholds fixed before held-out evaluation, paired comparisons from replayed histories, separate construction, certification and evaluation data, and evaluation across two environments with both cloned and RL-trained policies. Reporting certification cost and the negative result for a contextual activation gate also helps in judging practical applicability.
Critical weaknesses
  • The scope of the “certificates” needs clarification. Skills are certified individually but deployed together, where they may interact. The held-out evaluation tests the combined set, but the certification procedure does not cover those interactions. The statistical bounds also need care: Table 10 reports perfect lower bounds after roughly 30 games with no observed failures, although a small failure rate remains possible. Clearer limits on the claimed assurance, checks for skill interactions, and bounds that account for unobserved failures would strengthen the paper.
  • The deployment savings take substantial time to repay the upfront testing cost. On ALFWorld, PRA saves fewer than two decisions per episode, requiring approximately 42,000 episodes to recover certification costs. This limits its usefulness for agents that are updated frequently or deployed for relatively few tasks.

NB: The presentation could make the method’s simplicity and intuition more accessible. Readers encounter several new terms—promotion, admission, binder, commit, catalogue and fidelity—alongside concepts from options, STRIPS effects and non-inferiority testing. Introducing these through a single worked example, and retaining only the terminology needed to explain the method, would help. This is a clarity concern rather than a limitation of the underlying approach.

Questions for authors

Would a simpler selection rule perform similarly—for example, retaining skills that reduce average decision counts while keeping the observed success-rate drop within the same tolerance, using the same evaluation budget? This comparison would help clarify the added benefit of PRA’s confidence bounds.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 22t5

Following your suggestion, we compared the simpler rules using the same candidates and certification records. The results differ by environment. The mean rule is competitive on ALFWorld. On WebShop, PRA saves more decisions than PRA-Mean at similar fidelity, while the effect-free mean selector saves slightly more than PRA but has substantially lower fidelity. The table below reports these outcomes separately so that greater compression is not mistaken for more reliable action execution.

Simpler selection, at the same budget

Comparison performed. PRA-Mean replaces the submitted lower bounds with means. PRA-Mean w/o effect also removes the effect condition, requiring only positive mean saving and mean success loss within tolerance. Each catalogue is tested directly and after the same joint check. A failed check returns that seed to Flat. The table gives held-out means after the nominal check, including Flat outputs (Figure 2; Table 5):

Panel / selectorSuccess differenceDecisions savedFidelity
ALFWorld: PRA+0.0001.680.886
ALFWorld: PRA-Mean+0.0022.440.852
ALFWorld: Mean w/o effect−0.0071.770.808
WebShop: PRA+0.04014.720.949
WebShop: PRA-Mean+0.01310.160.950
WebShop: Mean w/o effect+0.02715.470.763

What the results show. On ALFWorld, PRA-Mean saves more decisions at similar task success. On WebShop, PRA saves 14.72 decisions versus 10.16 for PRA-Mean at similar fidelity (0.949 versus 0.950). The effect-free mean selector saves 15.47 decisions, but its fidelity is 0.763, with 265/1,118 aborted commits versus 35/681 for PRA. Thus, greater decision savings alone do not establish that an action reliably realizes its declared effect. PRA's effect condition checks this property alongside the separate checks on savings and task success.

Table 5 includes paired bounds and the submitted bootstrap selector. Before the joint check, effect-free ALFWorld selection has success difference −0.025 with lower bound −0.055, so non-inferiority at the 0.05 margin is not established. One seed returns to Flat, giving −0.007 above. That distinction separates candidate selection from the final deployment decision.

The admission checks (W1)

Screening change. Because the percentile bootstrap cannot represent a failure that was never observed, we replaced it with cluster-adjusted Korn–Graubard fidelity and paired Tango success bounds. Success is evaluated on Eq. 2's per-episode scale. We then examined coverage in simulation rather than assuming the replacement resolves the statistical issue. Appendix D still shows undercoverage with correlated commits and failure-dependent commit counts, so we describe these as nominal screens.

Catalogue check. Candidate-wise screening does not establish the behavior of the combined catalogue. We therefore freeze the catalogue and dispatcher, then compare them with Flat on 256 unused tasks. This additional check rejects a submitted-rule WebShop catalogue with 3 gains and 16 losses, returning it to Flat despite its entries passing individual admission. To examine sensitivity to the approximate success bound, we also report a conservative exact bound for paired binary success, which gives lower deployment rates and savings. This does not make fidelity or bootstrap savings exact. No pruning or re-testing uses the same joint block.

Cost and explanation in the revision (W2)

Including the joint check, break-even is 46,337 ALFWorld episodes for the cloned policy and 81,556 for SkillRL-7B. The cloned WebShop estimate is about 2,100 sessions. These identify a stable, repeatedly deployed policy as the economic setting (Appendix L). Section 4 now follows one “clean” example through screen, freeze, check and deploy, with representation details moved to Appendix A.

AuthorsOpenReview ↗

Follow-up on the simpler controls and revised procedure

Thank you again for your time and thoughtful review. We hope our response has clarified what the individual checks contribute. The added comparisons and procedural changes address three parts of that question.

  • First, to test whether simpler selection suffices, we evaluated mean-based rules at the same evaluation budget. The mean rule is competitive on ALFWorld. On WebShop, PRA gives 94.9% fidelity versus 76.3% for the effect-free mean selector, while saving 14.72 versus 15.47 decisions. This comparison illustrates the distinction between reliably executing the declared action and maximizing decision savings.
  • Second, because the zero-failure bootstrap bounds were not justified, we replaced them with nominal cluster-adjusted fidelity and per-episode success bounds. Coverage simulations still show fidelity undercoverage in some settings, so we do not call these exact guarantees. To assess the deployed combination as well, we added a separate unused-task check of the frozen catalogue and dispatcher, returning a failing seed to Flat.
  • Finally, we included that check in the cost calculation. The cloned-policy break-even is about 46,000 ALFWorld episodes and an estimated 2,100 WebShop sessions. This makes repeated deployment of a stable policy the relevant economic setting.

We would be happy to discuss any part of the comparison or admission procedure that remains unclear.

AuthorsOpenReview ↗

Thank you for considering our response

Dear Reviewer 22t5,

We noticed that you raised your rating, and we’re grateful for your recognition of our work.

Following your suggestions, we added a comparison with simpler selection rules in Section 5.3 and Table 5. The revised paper also describes the joint check for the full catalogue in Section 4.4.

Thanks again for the time and care you put into reviewing our work!

Best regards,
The authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response: policy-relative admission and joint deployment checks

Thank you for the reviews. The new comparisons strengthen the distinction between a sequence that executes reliably and a sequence this particular policy should use. The same option-then-buy entry improves the cloned policy's certification success but harms SkillRL-7B's, so it is admitted for one and rejected for the other.

The procedure now tests the deployed catalogue.

To test the combination that will actually be deployed, we added a final joint check. After screening candidates, we freeze the catalogue and dispatcher, then compare them together with Flat on unused tasks. A failing seed returns to Flat, and we include those returns in the held-out results in Table 1. This directly addresses catalogue composition. Separately, we corrected the treatment of the zero-failure bootstrap boundary.

The added comparisons show the trade-off.

The simpler policy-tested controls are informative. A mean-based selector can save more on ALFWorld. On WebShop, PRA achieves fidelity of 0.949 versus 0.763 for the effect-free mean selector, while saving 14.72 versus 15.47 decisions. We now state this trade-off explicitly.

The revised PDF includes the nominal procedure, an exact-success-bound sensitivity, coverage simulations and costs including the joint check. The nominal bounds are not finite-sample guarantees. Sections 2–4 also clarify the planning precedents and explain the procedure through one running example.

To help locate the revisions, we have marked key revised passages and captions in gold-brown.

AuthorsOpenReview ↗

Response Summary: Execution Fidelity, Policy Value and Joint Admission

Dear Reviewers,

We appreciate your recognition of the paper's distinction between constructing an executable skill and deciding whether a policy should use it. The original ALFWorld result makes this distinction concrete: catalogues with over 90% execution fidelity can reduce task success from 0.861 to 0.493. PRA therefore evaluates both the declared effect and downstream value for the policy that will deploy the catalogue.

Your questions concern how this admission decision is justified and when it is useful. We bring the main answers together below.

Individual and joint admission — 22t5 and 1bKv. Individual tests cannot establish the performance of a combined catalogue. We added an independent check of the frozen catalogue and dispatcher on 256 unused training tasks per seed, with a return to the primitive policy if it fails. There is no pruning and repeated testing on that sample. We also replaced the degenerate candidate-level bootstrap bounds and reported calibration and exact-test sensitivity separately. This extends admission to the deployed combination, on the stated certification distribution.

What do simpler selectors reveal? — 22t5 and kkbv. The new controls distinguish compression from faithful execution. On WebShop with the cloned policy, the effect-free mean selector saves 15.47 decisions at 76.3% fidelity, whereas PRA saves 14.72 at 94.9% fidelity. These held-out results support the need to evaluate both properties. We also distinguish the specific policy-relative admission setting from the macro-utility selection already studied in Macro-FF and Wizard.

When does admission pay off? — 1bKv and kkbv. We report certification cost and the deployment volume needed to recover it, alongside policy-specific purchase attribution. The intended benefit is repeated use of a frozen policy. The same routine can be helpful for one policy and unsuitable for another, which is why PRA tests it against its actual deploying policy.

We hope these answers make both the original finding and the admission checks easier to assess. Further questions about the joint check or deployment setting would be very welcome.

AuthorsOpenReview ↗

Summary of Major Contributions, Strengths, and Responses to Concerns

Dear AC, SAC, and PCs,

We appreciate your work in managing the review process and the reviewers’ careful reading of our paper and rebuttal. During the discussion, 22t5 raised their rating from 2 to 3, and 1bKv read our response and confirmed their weak-accept recommendation. We are grateful for the time taken to revisit the evidence. To assist your decision, we summarize the contribution, the strengths identified in the reviews, and the results addressing the main concerns.

1. Core Contributions

  • Policy-relative admission of skills. PRA asks whether a constructed routine should become an action for the particular frozen policy that will use it. It combines fidelity to the declared effect with decision savings subject to a task-success constraint, providing a way to select useful actions without retraining the policy.

  • Evidence that reliable execution does not establish deployment value. In the original ALFWorld ablation, effect-only admission achieves 94.5% execution fidelity, yet task success falls from 86.1% to 49.3%. This demonstrates why effect fidelity and downstream policy value must be assessed separately.

  • Evidence that admission depends on the policy. The same routine can help one policy and harm another. On WebShop, the cloned policy admits the option-then-buy routine while SkillRL-7B rejects it. Catalogue-transfer experiments further show why admission should be tied to the policy that will deploy the skills.

2. Strengths Recognized by Reviewers

  • A clear and useful deployment question. 1bKv recognized the practical contribution of deciding which routines to enable without retraining the agent. 22t5 and kkbv highlighted the evidence distinguishing executability from value.

  • Controlled evaluation of policy dependence. kkbv emphasized the policy-dependent admission and catalogue-transfer experiments. 22t5 also noted the fixed thresholds, paired comparisons, separate data splits and evaluation across environments and policy families.

3. Responses to Key Concerns

The final manuscript supplements the original admission experiments with direct tests of catalogue interactions and simpler selection rules, alongside statistical and cost analyses.

ReviewerConcernEvidence in the final manuscript
22t5, 1bKv, kkbvCan individually admitted skills be harmful together?We test the frozen catalogue and dispatcher jointly on 256 unused tasks per seed. For WebShop seed 42, the check rejects a PRA-Bootstrap catalogue with 3 gains and 16 losses, while the PRA catalogue passes with 29 gains and 11 losses (Table 4).
22t5Is the admission rule better than a simpler selector?At the same evaluation budget and nearly identical fidelity on WebShop, PRA saves 14.72 decisions per session versus 10.16 for PRA-Mean (Table 5).
22t5, 1bKv, kkbvWhat assurance and deployment cost do the results support?Boundary-case calibration checks and exact-test sensitivity analyses specify the statistical scope. Costs and break-even analyses across all four panels make the conditions for recovering certification costs explicit.

4. Concluding Remarks

PRA provides a concrete procedure for deciding which skills a frozen policy should use, supported by experiments showing why construction, faithful execution and policy value are distinct. The joint-catalogue and selector comparisons strengthen the evidence for that decision procedure. We are grateful for the constructive feedback and your consideration of these contributions.

The authors of PRA

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.