- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies sequential testing of a fixed policy library when executing an unverified candidate can itself violate a safety constraint. JACE maintains time-uniform risk bounds, mixes candidate episodes with a known-safe baseline, prioritizes unresolved candidates, and stops when it can certify a recommendation satisfying the risk limit and matching the frozen utility of all candidates safely below the limit by a prescribed margin. The paper proves joint safety of data collection and deployment, derives upper bounds on the number of episodes needed to complete the decision, and gives a worst-case lower bound with matching parameter dependence up to logarithmic factors. Experiments on PointGoal1, CarPush1, and synthetic libraries examine completion cost and when prioritization is useful.
Critical strengths
The paper addresses the right combined problem rather than treating safe exploration and stopping independently. Controlling risk during data acquisition while still obtaining enough evidence to rule out higher-utility competitors is genuinely different from merely identifying one safe candidate. The matching upper and lower analyses give the resulting cost decomposition real theoretical weight.
The experiments support the mechanism and do not only showcase the favorable case. PointGoal1 exhibits a sizeable reduction in completion cost, while CarPush1 demonstrates how a near-threshold higher-utility competitor can make completion much harder than initial certification. This is useful evidence that the authors understand the limits of prioritization rather than presenting it as uniformly superior.
Critical weaknesses
The utility guarantee is deliberately weaker than selection of the best safe policy. Because the comparison class excludes policies near the safety boundary, JACE may omit a higher-utility policy that is in fact safe. The paper is transparent about this, but the practical consequence can be significant if utility varies sharply in the margin band.
The theory relies on a relatively restrictive deployment model. The policy library, utility scores, episode laws, thresholds, and baseline guarantee are frozen before online collection. In many realistic settings, both utility estimates and environment risk drift, and obtaining a sufficiently certified fallback policy can itself be expensive. The current guarantees do not directly cover those cases.
Questions for authors
How should a practitioner choose the safety margin when the utilities of policies near the risk threshold differ substantially? Can you report a utility-versus-completion-cost curve rather than fixing a single margin?
Do the reported sample-efficiency comparisons include the episodes needed for baseline qualification and library preparation? If not, how does the ranking change when all data-collection costs required by the procedure are included?
Which parts of the guarantee survive if utility estimates are updated online or if policy risk changes slowly over time?
3 comments
W2 + Q2. How do preparation costs and historical data affect the ranking?
On PointGoal1, JACE and Safe-RR averaged 5,412.8 and 8,615.5 online episodes, respectively. Adding the same preparation cost preserves that ranking but reduces the relative saving. Actually reusing candidate-risk observations can instead eliminate differences between allocation rules.
| PointGoal1 cost accounting | JACE savings relative to Safe-RR |
|---|---|
| Online collection only | 37.17% |
| One deployment omitting separate assessment of the other 15 policies | 1.45% |
| One deployment with all preparation | 0.62% |
| Ten deployments sharing all preparation | 5.35% |
Pilot selection, scoring and baseline qualification used 32,000, 160,000 and 20,000 episodes. PointGoal1 pilot counts use archived summaries. The 1.45% scenario charges these 212,000 episodes and omits the benchmark's 300,000 assessments of other policies. Public offline-data acquisition and training costs were unmeasured and excluded. With a more conservative baseline risk cap, 40 paired replays still gave JACE a 36.9% online saving over Safe-RR (Appendices C.8–C.9).
With 10,000 historical episodes per policy and ten paired future runs, all four methods finished immediately on PointGoal1 and had identical CarPush1 costs on each seed, averaging 15,801.8 additional episodes. With ten disjoint 1,000-episode blocks per policy, CarPush1 means ranged from 22,338.4 to 22,366.7. Every method improved over its cold start in every pair and completed 10/10 with independent assessment support. These blocks share frozen development scores and measure sensitivity to the reused history. Table 7 reports fully observed settings. Appendix C.8 bounds missing costs for incomplete 100-episode-history replays.
Valid reuse continues each policy's risk test from its complete historical sequence under the original error budget. Policies, tests and error allocations are fixed before the first retained observation, sampling decisions precede outcomes, and independent policy streams retain their i.i.d. laws. Rewards from those same episodes can supply the scores even when risk and reward are dependent (Appendix B.4).
Supplemental CarPush1 assessment used 1,716,537 episodes, including 65,237 baseline episodes. The sequential procedure specified before collection gave fixed-output expected-return loss bounds of 0.011480 for the margin set and 0.308232 for the full-safe set. The terminal test was designed afterwards on these data and is exploratory, not a joint 0.05 certification of both 0.01 targets. The sequential bounds remain reusable for unchanged policies, output, targets and episode laws (Appendix C.9, Table 10).
W2 + Q3. What survives changes in scores or risks?
When risks are stationary, updating scores using observations already collected preserves both collection and recommendation safety. Whenever the stopping test passes on joint risk coverage, the recommendation's current score matches or exceeds those of the baseline and all margin-safe policies. Later score changes can reopen the search, so permanent completion and the original time bound need additional assumptions. Expected-return quality also requires score-error control or direct assessment (Corollary A.4, Appendix B.8).
Our experiments use stationary risks. Appendix B.9 extends the guarantee over a fixed horizon when a known bound limits how far each policy's conditional risk can change between any two rounds. We fix the observation window in advance and use fresh randomization for each draw. For each policy, we widen the confidence interval based on recent observations by the average allowed drift since those observations. A policy with no observations in the window receives the full interval . Under adaptive sampling, all these intervals cover the current nonbaseline risks throughout the horizon with probability at least .
If the baseline also has a valid upper bound uniformly below the safety limit throughout the horizon, with failure probability at most , collection safety, recommendation safety and validity of each current margin comparison hold jointly with probability at least . With fixed scores, a recommendation can remain in force for a specified period if the bounds widened for that period still certify its safety and exclude every higher-scoring candidate from the margin-safe set. Once that certificate expires, sampling must resume and previously excluded candidates can reenter.
Summary of our responses
Dear Reviewer wtCN,
Thank you again for your time and questions. To facilitate your review, we briefly summarize our responses:
- Q1 (choosing the margin): Table 8 reports completion cost and returned score across margins. On CarPush1, widening the margin from 0.02 to 0.03 reduced JACE's mean capped cost by 23.1% and raised completion from 7/10 to 10/10, at a lower mean score. Existing held-out data already certify a CarPush1 policy within a 0.01 score tolerance with no new collection (Proposition B.2).
- Q2 (preparation costs): On PointGoal1, JACE's saving over Safe-RR is 37.17% for online collection only, 0.62% when one deployment bears all preparation and 5.35% when ten deployments share it. The ranking is unchanged.
- Q3 (changing scores or risks): With stationary risks, updating scores with collected observations preserves collection and recommendation safety. Appendix B.9 extends the guarantee over a fixed horizon when a known bound limits risk drift between rounds.
If our responses have resolved your questions, we would greatly appreciate it if you could consider updating your evaluation. We welcome any further questions.
Best regards,
The Authors
Response to reviewer on quality–cost tradeoffs and deployment assumptions
We sincerely thank the reviewer for asking how the margin affects deployment quality, how preparation changes the cost comparison, and which guarantees survive changes in scores or risks.
Choose the margin according to how close to the risk limit the score guarantee must extend. Even a narrow margin can omit a high-scoring safe policy. On CarPush1, widening the margin from 0.02 to 0.03 reduced JACE's mean capped cost by 23.1% and raised completion from 7/10 to 10/10. Two runs that completed at both margins returned a lower score at 0.03. All four methods reached the 40,000-episode cap at each of the three smallest margins. The table shows JACE on ten paired seeds (Appendix C.8, Table 8).
At 0.03, 6 runs scored 2.8004 and 4 scored 2.5091. At 0.02, 2 of those 4 hit the cap and 2 were among the 7 scoring 2.8004. On PointGoal1, all methods scored 16.6130 (10/10 per margin). JACE averaged 5,792.0 episodes at 0.004 and 5,372.0 at 0.03. Independent assessment supported all declarations.
When the target is the best safe policy, compatible historical data can settle the decision directly. Using simultaneous risk intervals for all policies, including the qualified baseline, subtract the highest certified safe score from the highest score still compatible with safety. This bounds the frozen-score loss of the selected policy. If none qualifies, we withhold certification. On CarPush1, the 320,000 existing assessment episodes gave an upper bound below 0.008674, meeting a tolerance of 0.01 with no new collection and joint safety or score error at most 0.005 (Proposition B.2).
For a cold start, let ϵ be the largest acceptable score loss. Suppose external evidence establishes that every safe policy scoring more than ϵ above the baseline has risk at least Δ>0 below the safety limit. Choosing a positive margin no larger than Δ preserves the score tolerance at exact completion on joint coverage, provided the certificates have the required resolution. The sufficient deadline must also fit the remaining budget. Without information separating risks near the boundary, this guarantee has no uniform finite deadline (Appendix B.6).
For a separately requested fresh run at margin 0.02, the sufficient deadlines are 37,716 episodes for PointGoal1 and 124,031 for CarPush1. The corresponding frozen-score loss bounds against the best safe policy are 0 and 0.3001 (Table 9). None of the tested CarPush1 deadlines fits 40,000 episodes. The direct historical check is at least as tight on the same data.
Proposition B.5 allows the planning data to guide the choice among a predefined finite list of configurations. Once a configuration is chosen, we fix it and collect fresh episodes independently of the planning data. The policies and scores remain fixed, the planning intervals must cover all risks simultaneously, and baseline qualification must remain valid after selection. Policy streams are independent, with the same i.i.d. episode law for each policy across stages. With error allowances of 0.005 for planning, 0.045 for the deadline conditional on valid planning information and 0.05 for safety and valid completion, the unconditional probability of issuing a promise and violating it is at most 0.10. Newly commissioned preparation and reserved validation also count toward the total budget.