acceptodds
Under review as a conference paper at ICLR 2027

Safety Certificates for Selective Distillation: Routing Entropy and Verifiable Adaptation

Abstract

Selective distillation leaves most histories without a teacher label. A safety certificate must therefore explain both what limits behavior there and what permits useful generalization. We study this problem using finite sequential models and architectures with enforced adaptation regions. Our main coverage result handles a base far from its teacher: under biased routing, the exact optimal supervision allocation reduces to weighted marginal gains, even when every router is eligible for supervision. At fixed fractional risk reduction, the budget exponent is routing entropy rather than the logarithm of the number of histories. We then construct gated logit envelopes that preserve a base outside a declared region while allowing large teacher-directed changes inside it, and derive parameter-only risk certificates. Experiments distinguish these structural guarantees from ordinary regularization. Four-action, two-layer transformers face rare teacher exceptions at horizons 24 and 32; query targeting, coarse sharing, and bounded residuals are varied separately. Fixed-multiplier controls on held-out tasks show that signed utility-aware local selection can match exact planning at matched completion and supervision budget. The practical recommendation is to establish the adaptation region, count rare-label access, include utility in selection, and check any behavior outside the certified envelope.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
selective distillation, safety certification, routing entropy, adaptation regions, supervision complexity

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Q4e8OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  • The paper studies safety certification for selective distillation with limited teacher supervision.
  • It develops an entropy-based supervision allocation theory and a gated architecture with parameter-based safety guarantees.
  • Experiments demonstrate the importance of targeted queries, trusted adaptation regions, and utility-aware selection.
Critical strengths

The paper provides a framework that connects supervision allocation, architectural constraints, and practical safety certification for selective distillation.

Critical weaknesses
  • The paper combines many loosely connected components, making the central contribution difficult to identify.
  • Several technical results rely on highly specialized assumptions, and their relevance to realistic selective distillation remains unclear.
  • The experiments are limited to small synthetic environments and therefore provide insufficient evidence for practical safety claims.
  • The presentation is unusually dense and sometimes reads as an aggregation of arguments and caveats rather than a coherent research narrative.
Questions for authors
  • Could the authors clarify the central contribution and explain how the theoretical, architectural, and empirical components form a unified framework?
  • Which assumptions are essential to the main theoretical results, and to what extent can they be relaxed in realistic selective-distillation settings?
  • Could the authors provide experiments on larger-scale or real-world tasks to demonstrate the practical relevance of the proposed safety certificates?
  • Could the authors streamline the presentation and more clearly distinguish the main claims, supporting results, and limitations?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Q4e8 (1/2)

Response to Reviewer Q4e8 (1/2)

Thank you for your review.

Q1 and W1: the central contribution. One question ties the parts together: which resource supplies the per-history behavior bounds that a certificate needs? The options are constraints (R1), a trusted gate (R2) and teacher queries (R3). A constraint marks a history where the certificate may assume teacher rather than base behavior.

The central contribution is Theorem 1 with Corollary 2, on R1. A base policy makes m binary decisions (routers), each taking action 1 with probability at least ℓ > 1/2, and then enters one of 2^m identical copies of a short failure-prone stage (a hazard), while the teacher is far from the base. Theorem 1 shows that an optimal placement of B constraints puts none on routers and fills hazard decisions in decreasing order of route probability times risk reduction. By Corollary 2, at a fixed fractional risk reduction, log₂ of the required number of constraints grows asymptotically by h₂(ℓ) bits per router (h₂ is the binary entropy), instead of 1 bit under symmetric routing.

A gate is a parser-computed indicator of the histories where the student may adapt. By Proposition 3, a gated network keeps the base outside the gate and allows large teacher-directed changes inside it, so a trusted gate can replace per-history constraints. Teacher queries are still needed, because each cell of the gate, a class of histories sharing one learned logit offset, can be fitted only if labels reach it (Section 4). In Section 3's example (m = 8, hazards of k = 3 decisions), new Appendix E shows that a three-cell gate with offsets at the teacher's logits and no residual certifies p³ = 0.125, where p = 0.5 is the teacher's harmful-action probability. Per-history constraints (Eq. 1) reach this value only if all 768 hazard contexts are constrained to follow the teacher exactly. The price of the gate is trust in it, labels in each cell, and a teacher that behaves the same at each hazard step in every copy. Without a trusted gate, Theorem 1 still decides where constraints go.

Each experiment isolates one question. Section 5.1 varies how teacher queries are targeted (R3), for a learner with a correct gate (R2) and for one without a gate, and Section 5.2 measures what a too-coarse gate costs in fit and certificate tightness. Section 5.3 does not test Theorem 1, which optimizes a risk-only certificate. For one family of small trees and budgets up to 8, it shows that selection must account for completion: at 95% completion and B = 8 constraints, a rule that scores histories by their signed effect on risk and utility has 4.57% risk, against 17.35% for the teacher-cost rule, which ignores completion (Table 3). The submitted l.448–449 wrongly suggested that the experiments test Theorem 1's assumptions, and Section 1 now states how each part relates to the main result.

Q2 and W2: which assumptions are essential. Two are essential: identical hazards, and a bias toward action 1 shared by the routers, or at least by those at the same depth. Without either, supervising a router can be strictly optimal. Such structure arises when many earlier routes lead to the same risky protocol step (Section 3), but we do not claim it for general language-model tasks. Section 3 now says which proof step uses which assumption. Theorem 1 assumes that (i) all hazard copies are identical, (ii) a supervised router's interval contains 1/2, (iii) every unsupervised router has the same interval [ℓ, r] for the probability of action 1, with ℓ > 1/2, (iv) every hazard decision has the same caps on its harmful-action probability, v without and u < v with a constraint, and (v) the intervals at different histories can be chosen independently (rectangularity).

  • (i) is essential (counterexample in Appendix F). At the budget that removes half of the reducible risk (the gap between no and full supervision), Theorem 1's allocation with averaged parameters is on average 0.24–5.63% above the optimum across seven variants with heterogeneous hazards, while greedy allocation by each copy's own marginal gains stays within 0.5% on average (Table 7).
  • (ii) holds in Section 3's example, where the teacher's router probability is 0.5, and is used only to delete router constraints. Without it, Theorem 1's allocation is still optimal among placements that leave routers unsupervised (Proposition 8(c) and the paragraph after the proof of Proposition 7, Appendix F).
  • (iii) can be relaxed to one bias per depth. Theorem 1 then stays exact, and log₂(budget)/m differs from the average routing entropy per depth by O(m^−1/2) (Propositions 5 and 6).
  • (iv) can be relaxed to caps that vary by hazard step but not across copies, and Theorem 1 and Corollary 2 then hold in full (Proposition 7).
  • Without (v), for example when the student must use one action distribution at several histories, the backward-induction certificate is still a valid upper bound but need not be exact.
AuthorsOpenReview ↗

Response to Reviewer Q4e8 (2/2)

Response to Reviewer Q4e8 (2/2)

Q2 (continued). Without (i)–(iv) but with rectangular intervals, the tree-knapsack algorithm of Section 2 still finds the exact optimum in O(|V|B) operations on an explicit tree with |V| nodes, which grows exponentially in m. The gate guarantee (Proposition 3) needs none of (i)–(v). The gated certificate's cost grows with the number of safety states (states of a declared finite-state model that tracks failure) rather than with the number of histories, although the number of safety states can grow exponentially in the number of tracked protocol variables.

Q3 and W3: larger-scale or real-world tasks. We agree that this is the main limitation of our evidence: we have not run a study with real tools or pretrained models. Our certificate needs an explicit failure predicate (a program that decides whether an episode failed), a declared finite-state model and, for Section 4, a trusted gate. No benchmark we know of supplies all three. Tool-agent benchmarks such as τ-bench and AgentDojo come closest but would first need a verified parser. Synthetic settings supply all three, so the certificate can be compared with an exact or well-estimated risk. For example, a certificate that assumes the student stays within 0.01 nats of the base at unselected histories is above the observed risk for only 79.31% of 1,728 trained policies (Section 5.5).

Model size affects how closely the student fits the teacher, but not the certificate's validity or cost: the certificate uses only the gate, the base, the cell offsets and the residual's width, and Proposition 3 and its bounds (Eq. 7) hold for any residual network bounded by tanh, including a transformer over the full token history (l.170–171). As a step toward larger tasks, new Appendix I implements a synthetic protocol with eight tools, 48 calls per episode, and tool results that can be garbled in transit. Its parser agrees with the protocol specification on 1.58M stream prefixes. With 512 teacher labels, a 0.21M-parameter student, five times larger than our earlier models, has risk 20.51% (teacher: 20.59%) and a certificate of 28.54%, computed in about 16 ms. When results can be garbled, the certificate stays valid if the verifier tracks both the parsed and the true state and knows the garbling rate, and it stays useful only if the parser fails closed, reading a garbled answer to an irreversible call as a success (the riskier reading). This shows that the certificate's ingredients can be built and checked for a concrete protocol, not that the guarantee transfers to real tools or pretrained models (Section 7).

The abstract's "practical recommendation" (l.25–27) and "Practical steps" (l.460–468) were also stated too strongly, and both now apply only where the failure predicate is explicit and the gate can be verified.

Q4 and W4: streamlining, and separating claims from caveats. We cut repeated sentences and a proof sketch, moved the detail of Section 5.4 to Appendix C.4, and reduced Appendix D's local selection result to a two-sentence note in Section 5.5 that calls it a supporting approximation. Section 1 now has a table of the paper's seven main claims that says where each is supported, what it assumes and what is not claimed. Row 1 is the main result, rows 2–4 are supporting theory, and rows 5–7 are experiments. Section 7 has a labeled paragraph on limitations that points to Appendices F, G and I, and the general response lists the other presentation changes. Even after these cuts, the claims table and new summaries make the main text about one page longer than the submission, within the 10-page limit.

Reviewer_7QrLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies safety certification for selective distillation in finite sequential models. Its main result gives an exact supervision allocation for biased routing over repeated, identical hazard subproblems, including settings where routing nodes are eligible for supervision. An exchange argument and monotonicity reduce the adaptive worst-case problem to weighted marginal-gain allocation, yielding an entropy exponent for the context-constraint budget at a fixed fractional risk reduction. The paper also constructs gated logit envelopes that permit substantial adaptation inside a declared region while controlling changes outside it. Combined with a trusted finite-state interface, these envelopes enable risk upper bounds. Synthetic transformer experiments, finite-tree selection comparisons, and recurrent-policy diagnostics examine label access, certificate conservatism, and the validity of behavioral envelopes.

Critical strengths
  1. An exact structural result within a clearly specified model.

Under the routing and repeated-hazard assumptions of Theorem 1, there exists an optimal supervision allocation that leaves all routers unsupervised. The exchange and monotonicity arguments show that the allocation obtained from fixed routing weights also attains the adaptive worst-case optimum. This reduction is a substantive theoretical contribution beyond the classical marginal-gain allocation step and is the strongest consideration in favor of acceptance.

  1. An explicit architecture for enforcing behavioral bounds.

Given a trusted adaptation gate, Proposition 3 permits large changes inside the gate while bounding drift outside it; setting the outside residual width to zero preserves the base exactly. The logit bounds provide action-probability intervals without evaluating the residual network at every history. Combined with the declared finite-state model, these yield a risk upper bound. Table 2 usefully illustrates how accurate imitation can coexist with a loose certificate when the parameter cells omit relevant information.

Critical weaknesses
  1. The main result has a restricted scope, and the presentation makes its overall significance difficult to assess.

The exact allocation result relies on identical hazard subproblems, a common one-sided routing bias, and rectangular local uncertainty. The paper acknowledges these assumptions and explains that the architectural and experimental components serve different purposes. Nevertheless, following the changes in models, supervision resources, and objects being verified requires substantial effort. A concrete worked example explaining how these components relate would help readers assess the insight provided by the paper as a whole. My reservation concerns the breadth and presentation of the contribution.

  1. The scope of the finite-task evidence is insufficiently specified.

Section 5.3 reports a comparison on eight held-out tasks with equal teacher/task access. However, the task depth, count, and reporting split do not explain how terminal failure/completion labels and teacher/base policies were constructed. These choices determine the risk/completion trade-offs on which the selection rules are compared. Without this specification, it is difficult to assess how broadly Table 3 supports the recommendation of signed utility-aware selection as a competitive simple baseline for known finite tasks.

Questions for authors
  1. Please describe the construction of the terminal failure/completion labels and teacher/base policies in Section 5.3. What kinds of risk/completion relationships does this construction produce, such as improving both quantities or improving one at the expense of the other? How do the generation rules for the four tuning tasks relate to those for the eight reporting tasks? These construction details would help interpret the scope of Table 3's result for known finite tasks.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 7QrL (1/2)

Response to Reviewer 7QrL (1/2)

Thank you for the precise summary.

W1: a worked example that links the components. Appendix E is the worked example you asked for. It follows one hazard shape, three consecutive harmful actions, through Sections 3–5.2 and three resources: per-history constraints (R1, histories where the certificate may assume teacher behavior), a trusted gate (R2), and teacher queries (R3). Its lesson is that per-history constraints stay exponential, a trusted gate replaces them at the price of trust and labels in each cell, a coarse gate costs tightness but not validity, and utility must enter selection once completion matters. Those sections share only the hazard's shape, and step 6 (Section 5.3) leaves it, so we do not claim that one model satisfies every theorem. The steps below are the rows of Table 6, and only the gate calculation in step 2 and the box interval in step 3 are new.

Step 1 (Section 3, resource R1, exact certificate). Here, m routing decisions (routers) lead into 2^m copies of a hazard of k = 3 decisions, and every unselected router chooses action 1 with probability at least ℓ > 1/2. Let v and u be the caps on the harmful-action probability of a hazard decision without and with a constraint. With base probability 0.8, teacher probability 0.5, KL radii 0.01 and m = 8, the targets v²u, vu² and u³, which are the certificates when every copy has one, two or three constrained decisions, need 52, 191 and 768 constraints, against 256, 512 and 768 under symmetric routing (l.140–142). Some optimal placement puts no constraint on a router (Theorem 1), and at a fixed fractional risk reduction, log₂(budget)/m tends to the routing entropy h₂(ℓ) as m grows (Corollary 2). The count is still exponential in m, and the endpoint u³ needs all k·2^m constraints.

Step 2 (Sections 3 and 4, resource R2, gated certificate). In the same model, a trusted gate turns on after routing and has three cells, for 0, 1 or 2 consecutive harmful actions so far, each with one learned logit offset. If each offset equals the teacher's logits and the bounded residual is off (width zero), every hazard decision takes the harmful action with the teacher's probability p = 0.5, and the certificate is p³ = 0.125. The per-history certificate of Eq. (1) reaches this value only if all 768 hazard contexts are supervised and assumed to follow the teacher exactly. With radius 0.01, those 768 constraints give u³ ≈ 0.186.

Step 3 (Section 5.1, resources R2 and R3, parameter certificate against estimated risk). The gate is the critical state, two consecutive EXECUTE actions, with one learned offset per prompt flag. At frequency 1/256 with 128 targeted queries, the certificate is 12.92% against a risk of 8.46%, and the teacher's risk is 8.91%. The gap comes from the logit box: the residual can move each logit up to γ₁ = 0.1 from the cell's offset, and the certificate must allow all such logits. The network matches the teacher's execution probability of 0.02 at the gate, but the certificate must use the upper end of the cell's execution interval [2.00%, 2.95%] (Appendix G.2, whose frequency-1/64 runs share these networks).

Step 4 (Section 5.1, resource R3 missing, estimated risk). Labels must still reach the cell. When sensitive prompts have frequency 1/256 and 128 queries fall at random stages of student rollouts, no learner receives a label at the critical state, and the risk stays near 98% (Table 1). Targeted querying needs 15,870 prompt draws on average to collect 128 labels for the gated student, and 31,400 for an ordinary transformer.

Step 5 (Section 5.2, a gate that is too coarse, parameter certificate). If the gate ignores a mode that the teacher depends on, fitting both modes needs a residual width above the threshold of Eq. (8), about 0.46 here. For the safer mode, the bound is then 82.96% at width 0.5 against a risk of 26.24%, because the shared box admits the other mode's behavior. With the mode in the cells, the bound is 38.00% at width 0.1 with the same label budget, against a risk of 27.08%. Both gates fit the teacher closely (teacher KL below 10⁻⁴ nats), so they are compared at similar fit rather than at equal width.

Step 6 (Section 5.3, selection objective, exact risk without a certificate). With the task fully known, Section 5.3 compares the exact risk of policies at 95% completion and shows that, for this tree family and budgets up to 8, risk-only selection is not enough once completion matters. This step does not test Theorem 1. Section 5.3 now states two limits of Table 3. First, the exact scalarized planner minimizes R − λU, risk minus λ times completion, so it is a reference planner rather than an oracle. Second, at B = 16 the risk-only linear rule has lower risk at 95% completion (1.11%) than the exact scalarized planner (1.77%) and the signed rule (1.82%).

AuthorsOpenReview ↗

Response to Reviewer 7QrL (2/2)

Response to Reviewer 7QrL (2/2)

W1: the scope of Theorem 1. Theorem 1's exact allocation, which supervises no router, needs identical hazards, one bias shared by all routers or by those at each depth, and rectangular intervals. Section 3 now says which proof step uses each. Outside that scope, the tree-knapsack algorithm of Section 2 still finds and certifies the optimal placement on an explicit tree with rectangular intervals, and Proposition 3 needs only a base that gives every action positive probability. In a new example in Appendix F, one router leads to two hazards with different caps. With one constraint, the certificate is 0.54 when the router is supervised, 0.73 or 0.815 when a hazard is constrained, and 0.82 without the constraint, so supervising the router is optimal. Our reply to Q4e8 (Q2) says how far each assumption can be relaxed, and our reply to KMMc summarizes the sensitivity study (Table 7).

W2 and Q1: how the Section 5.3 tasks were built. The tasks are complete binary trees of depth 5 with 32 leaves. Each task draws its leaf labels once: a leaf fails with probability 0.35, every failing leaf completes, and a safe leaf completes with probability 0.75, so failure and completion are positively correlated (0.32 by design). With c the completion label and y the failure label, the teacher is a softmax at temperature 0.25 over its continuation value of c − 4y. The scripted base uses the same rule on completion alone, so it ignores failure. Each task has two bases, 4,097-parameter GRUs fitted to the scripted base, which differ by at most 0.003 in any conditional, so the eight held-out tasks are the unit of replication. New Appendix H specifies this construction, which the submitted paper omitted, and rerunning it from the saved bases reproduces Table 3 exactly. Section 5.3 now summarizes it and defines the quantities in the signed score, Eq. (9).

The tuning tasks (seeds 1000–1003) and the held-out tasks (seeds 1004–1011) come from the same generator and differ only in seed. Table 3 uses the fixed multiplier λ = 1 and, for teacher-cost, exponent 1 on the teacher's visitation probability. In a separate tuned variant (Table 12), each method except the risk-only rule picks one grid parameter per budget on the tuning tasks, either λ or, for teacher-cost, the visitation exponent. The comparison is retrospective. λ = 1 and the split are recorded in the experiment's configuration but were not pre-registered, and earlier versions of this work had already analyzed all 12 tasks.

Risk and completion relationships. On every held-out task–base pair, the teacher lowers risk at a small completion cost: its risk is 0.53–1.55%, against 33.39–53.12% for the bases, at a cost of 0.36–1.09 completion points. We also replaced the teacher's conditional with the base's at each of the 496 held-out contexts in turn, keeping the teacher elsewhere, and weighted each context by the teacher's probability of reaching it times the teacher–base difference. On 45.6% of this weight the teacher lowers risk without changing completion, and on 33.2% it improves both. It trades completion for lower risk on 16.2%, improves completion without changing risk on 3.2%, and improves completion at a risk cost on 1.8%.

The trade-off matters for policies that copy the teacher at only B contexts. At B = 8, the exact planner with λ = 0 (Table 13) reaches a raw risk of 4.43%, before mixing to 95% completion, by steering episodes from fail-complete leaves to safe-abort leaves, which neither fail nor complete. Its completion then averages 88.61% and is below 95% in 10 of the 16 task–base pairs. There, the mixture that reaches 95% completion must include the high-risk base, so the mean risk at 95% completion over all 16 pairs is 19.91%. With λ = 1, the exact planner and the signed rule keep completion at 99.51% (raw risk 4.78%) and reach 4.57% at 95% completion.

Other label regimes (Table 14). A label regime is the rule that assigns failure and completion labels to leaves. We reran the whole protocol on the eight held-out tasks and 24 new ones under the original regime (L0) and three new regimes (L1–L3). Completion is independent of failure in L1, failures never complete in L2, and in L3 failures are more frequent and safe completions scarcer. At B = 8, the signed rule stays within 0.3 points of the exact planner on average in every regime. Over all 32 tasks, its gain over teacher-cost is 7.67 points in L0, 6.62 in L1, 0.29 in L2 and 23.19 in L3. The gain is nearly zero in L2 because a base that seeks completion already avoids failures there. In L0 it is 5.97 points on the 24 new tasks, against 12.78 on Table 3's eight. Section 5.3 now states that the result is shown only for this tree family and that its size depends on the regime and the tasks.

Thank you for the detailed response. The task-generation details address my clarification question, and the worked example makes the connection between the components clearer. However, my remaining concern is the overall significance of the contribution. The main theorem relies on a narrowly structured routing and hazard model, while the supporting synthetic experiments provide limited evidence of applicability beyond the controlled settings studied. Although the revision improves clarity and strengthens the evidence, I remain unconvinced that the combined contribution is sufficient for acceptance. I will therefore maintain my score

AuthorsOpenReview ↗

Response to Reviewer 7QrL (follow-up)

Thank you for your follow-up and for confirming that the task construction and the worked example are now clear.

On significance, the paper shows when a safety claim about a selectively distilled student is valid. Common practice does not suffice: a small rehearsal loss does not keep the student close to the base model, and a certificate that assumes this closeness bounds the observed risk for only 79.31% of 1,728 policies (Section 5.5). Per-history constraints (Theorem 1) make the claim valid, and so does a trusted gate (Proposition 3), provided that labels reach each gate cell (Section 5.1).

The narrow routing and hazard model limits only the exactness of Theorem 1. In the sensitivity study, at a budget that removes half of the reducible risk, a greedy allocation stays within 0.5% of the exact optimum on average (Appendix F, Table 7). Proposition 3 needs none of the assumptions of Theorem 1 and holds for any residual network bounded by tanh, including a transformer over the full token history.

Beyond the controlled settings of Section 5, Appendix I applies the gated certificate (an upper bound on the student's risk) to a tool-call protocol with eight tools, where the student's risk is 20.51% and its certificate is 28.54%. Section 7 names real tools and pretrained models as the main next step.

Reviewer_KMMcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies safety certificates for selective distillation, deriving exact supervision allocation and an entropy-based budget law under biased routing. Gated logit constraints provide verifiable behavioral envelopes. Synthetic experiments examine rare-label access, teacher fitting, and certificate tightness.

Critical strengths
  1. The exact allocation theorem has technical merit and a clear proof strategy.
  2. Supervision budgets, rare-label access, and trusted structure are distinguished, avoiding unsupported guarantees from training loss.
  3. The experiments separate teacher fit from certificate tightness and candidly report limitations.
Critical weaknesses
  1. The main theorem has a narrow structural scope. Section 3, Theorem 1, assumes repeated hazards, common u/v caps, and one-sided routing bias. The exchange proof in Appendix A.1 relies on isomorphic subtrees. These assumptions are acknowledged, but usefulness under heterogeneous hazards remains unclear. A small sensitivity study would better establish the applicability boundary.
  2. Practical guarantees depend on a correct gate. Section 4 treats it as a “trusted interface,” while Section 5.1 supplies SAFE/SENSITIVE flags and the critical state. The experiments therefore do not address the difficulty of obtaining the correct gate. A concrete tool protocol and controlled gate-error experiments would better support the practical recommendations.
  3. Certificates can be too loose. Table 2, at γ=0.5\gamma=0.5, gives mode-1 risk of 26.24% but an 82.96% bound; at γ=2\gamma=2, the bound reaches 99.97%. Such bounds remain valid but offer limited decision value. Finer gates improve tightness, so the required trusted information and verification costs need clearer accounting.
Questions for authors
  1. Is the allocation useful under heterogeneous hazards?
  2. How would a real protocol verify its gate, and what happens with gate errors?
  3. How do verification cost and certificate tightness change with state-space size?
5 comments
AuthorsOpenReview ↗

Response to Reviewer KMMc (1/3)

Response to Reviewer KMMc (1/3)

Thank you for checking Table 2 in detail. We ran the studies you suggested.

W1 and Q1: is the allocation useful under heterogeneous hazards? Partly. The reducible risk is the gap between the unconstrained and fully constrained certificates. At the smallest budget that removes half of it, Theorem 1's allocation with averaged parameters is on average 2.5% above the optimum over eight variants (seven with heterogeneous hazards, one with router-specific bias), while a greedy allocation using each copy's own marginal gains stays within 0.5% on average in every variant (Appendix F, Table 7). When the target removes 90%, the optimum supervises routers in 65.5–97.0% of the instances of each heterogeneous-hazard variant.

The exchange step of the proof (Appendix A.1) needs isomorphic subtrees, selected router intervals that contain 1/2, and ℓ ≥ 1/2, where ℓ is the lower end of every unselected router's interval. The monotonicity step (Appendix A.2) uses identical hazards with common caps and a shared bias. Propositions 5–7 relax the last two (one bias per depth, caps that vary by hazard step), not identical hazards. In the counterexample of Appendix F, one router leads to a severe hazard and a mild one. With a budget of one constraint, supervising the router gives the lowest certificate, 0.54, against 0.73 or 0.815 for constraining a hazard.

Four results still hold. With rectangular intervals, which can be chosen independently at each history, the tree-knapsack algorithm of Section 2 finds the exact optimum, which may supervise routers, in O(|V|B) operations on an explicit tree with |V| nodes. Proposition 8 gives a sufficient (but not necessary) check that the greedy allocation is optimal among placements that supervise no router. If routers at the same depth share one bias, Theorem 1 stays exact, and log₂(minimum budget)/m differs from the average routing entropy per depth by O(m^−1/2) (Propositions 5 and 6). With caps that vary by hazard step but not across copies, Theorem 1 and Corollary 2 hold in full (Proposition 7). With a different bias at each router, neither proof step applies: in an Appendix F example with B = 2, the best placement supervises the root router and certifies 0.561, against 0.710 for the best placement that supervises no router.

Sensitivity study (Table 7). We perturb the example of Section 3 at m = 4, 6, 8 and 10, with two seeds, varying the hazard lengths, the caps, or the bias per router or per depth. At the smallest budget that removes a fraction β of the reducible risk, we compare the exact optimum, which may supervise routers, with (A) Theorem 1's allocation with averaged parameters and (B) the greedy allocation. Gaps are relative to the optimum. At β = 0.5, (A) loses 0.24–5.63% when only the hazards differ and 7.5–8% when the bias varies by router or depth. At β = 0.9, the mean gap of (B) grows to as much as 10.3%. The check of Proposition 8 confirms (B) in at most 4.0% of the instances of any heterogeneous-hazard variant at β = 0.5, so near-optimality has to be confirmed with the tree-knapsack algorithm. The two seeds agree within 0.5 points on every mean gap.

AuthorsOpenReview ↗

Response to Reviewer KMMc (2/3)

Response to Reviewer KMMc (2/3)

W2 and Q2: verifying the gate, and the effect of gate errors. A real protocol can verify its gate by checking a deterministic parser against an independently written specification automaton. Appendix I checks its parser on 1,576,804 stream prefixes, which include every sequence of up to four calls and random full-length streams, and model checking would make the check complete. Gate errors can break the intended gate's certificate, but the certificates of Appendix G that account for errors stay valid (details below). The submitted paper treats the gate as trusted (l.187–189), and Section 5.1 gives the learner the flag, so query access is studied separately from discovering it (l.304–306). Section 4 now summarizes Appendices G and I, and the claims table lists the known gate as an assumption of Section 5.1.

The protocol of Appendix I. A prompt carries a permission record, SAFE or SENSITIVE, and the agent then makes 48 calls to eight tools. DELETE and SEND are irreversible, and CONFIRM resets the safety state. Each call is answered with OK or ERR, and the answer can be garbled in transit. On SENSITIVE tasks, an episode fails if it makes an irreversible call while two successful irreversible calls are pending since the last CONFIRM. A deterministic parser counts these calls and turns the gate on at a count of 2. A parser that fails closed reads a garbled answer to an irreversible call as OK, the riskier reading, and one that fails open reads it as ERR. Of 5,000 randomly perturbed streams, all 4,334 malformed ones turn the gate on. Training and evaluation compute the gate with the automaton's transition, not the parser, so only the parser–automaton check above links this gate to the parser. The certificate is a worst-case backward pass over the nine pairs of true and parsed counts, and the verifier must know the OK and garbling probabilities.

With an exact parser and 512 labels, the 0.21M-parameter student with a typed gate (the gated architecture of Section 4) matches the teacher, with risk 20.51% against 20.59%, and its certificate is 28.54%. An unconstrained transformer of the same size has a risk of 0.81%, but it avoids irreversible calls throughout SENSITIVE episodes, making 3.8 per episode against 20.2 for the typed gate. Its on-policy KL divergence from the teacher is 1.53 nats per decision, and its labels cost 39,436 prompt draws against 4,233. When 5% of answers are garbled, a parser that fails closed keeps the certificate valid and useful, at 28.38% for a risk of 20.61%. With a parser that fails open, the risk rises to 63.25%, and the certificate that assumes an exact parser, 28.53%, is violated in every seed, while the certificate over pairs of true and parsed counts, 65.79%, holds. A shield, which blocks irreversible calls whenever the gate is on, has zero risk with the exact parser but cannot represent the teacher's 2% rate of irreversible calls at the gate. The protocol is synthetic, the permission record is a trusted prompt token, and which states belong in the gate remains a design choice.

Gate errors. If the verifier's state model tracks the failure predicate, the gate and the cell label, the certificate is valid for the gate as implemented (Proposition 9). Otherwise, Appendix G gives four certificates that stay valid as long as the state model does not depend on the possibly faulty gate. The hull certificate, which allows every output the deployed policy could produce at an affected state, holds for any error pattern but is nearly vacuous here, at 98.30%. A rate certificate needs the exact error rate or a bound on each step's error probability, and an episode-mixture certificate needs errors drawn before each episode, with a bound on their probability. A new first-error bound is the intended gate's certificate plus a bound on the probability that an episode contains any gate error, and it holds for any error pattern.

On the saved networks of Section 5.1, independent false negatives (missed gate visits) at rates of 5% and 20% raise the risk to 20.67% and 51.30%. The certificate of the intended, error-free gate, 12.92%, is then violated in every seed, while the certificate that uses the exact error rate, 24.93% and 52.94%, holds. At about the same per-visit error rate, which is what an audit of gate decisions measures, errors concentrated in a few episodes do less harm. At 5%, misparsed prompts, which leave the gate off for a whole episode, give a risk of 10.87%. Parser desynchronization, which leaves it off for the rest of an episode, gives 11.75%, against 21.65% for independent errors in a rerun (Table 9). No certificate whose assumption holds is violated. For desynchronization, the first-error bound gives 16.56% at 5% and 30.03% at 20%. With Appendix I, these results favor parsers that fail closed and audits that bound how often an episode contains a gate error.

AuthorsOpenReview ↗

Response to Reviewer KMMc (3/3)

Response to Reviewer KMMc (3/3)

W3 and Q3: verification cost and tightness as the state space grows. In our command grammar, where only execution changes the safety state, and with a fixed base, the certificate costs O(H·|S|), where H is the horizon and |S| the number of safety states in the declared state model, and it needs no network evaluation. Tightness does not improve with the number of states as such: a mode's certificate exceeds its risk by roughly how far its cell's logit box (offset ± γ₁) reaches beyond that mode's teacher, so an added state variable tightens it only when it lets that box shrink. Section 5.2 now separates the two sources of looseness in Table 2 and adds a short paragraph on cost, with details in Appendix G.

Looseness forced by the coarse cell. For residual width γ₁ ≤ 0.5, the bound is within about 1.3 points of the network's mode-0 risk, a floor that every sound certificate ignoring the mode must reach. Because the coarse cell, gate and base all ignore the mode, the network's mode-0 behavior is admissible on mode-1 histories. At γ₁ = 0.5, for example, the bound is 82.96% and the estimated floor is 82.00%.

Slack of the box. At γ₁ = 1, 2 and 4, wider than the 0.5 that already fits both teachers, the bound exceeds each network's mode-0 risk by 14–23 points, of which an estimated 11.5–16.4 are box slack. At γ₁ = 2, for example, the bound of 99.97% that you cite compares with a floor of 80.63%. When we replay every saved network, the largest execution probability observed at the gate averages 26.0–26.5% over seeds, whereas the box admits up to 40.5%, 83.1% and 99.6% at γ₁ = 1, 2 and 4. A sound certificate built from the network's actual outputs instead of the box would need to evaluate the network on about 1.3 × 10¹⁹ gate histories per mode at H = 32, which the parameter-only certificate avoids. At a nine-decision horizon, where enumeration is feasible, the exact output range of one γ₁ = 2 network gives 35.03% for mode 0, close to its estimated risk of 34.96%, while the box gives 85.62%. γ₁ should therefore be kept just above the smallest width that fits both teachers, about 0.46 here (Eq. 8).

Trusted information. Adding one observed bit, the mode, to the gate brings the bound to 38.00% at γ₁ = 0.1 against a risk of 27.08%, with the same label budget (l.320). Two targets in one cell need a residual width above D/4, where D is the spread across actions of the log-ratio of the two targets' probabilities (Eq. 8). An extra state variable that does not separate such targets therefore does not tighten the certificate and only adds label cost.

More modes (Table 10). With 2, 4 and 8 modes in one grammar, one coarse cell gives every mode about the same bound, 83–84%, while with one cell per mode each mode's certificate is 9–14 points above its risk. The certificates take between 0.6 and 5.9 ms because the backward induction has only H·|S| = 96 to 768 state-steps. What grows is the cost of reaching rare cells: 512 labels need between 765 and 2,214 prompt draws.

Costs that do grow. A learned base needs its interval evaluated on every history in a state, or bounded conservatively, and the state space can grow exponentially in the number of tracked protocol variables. Label cost grows because finer cells are rarer. If π_min is the smallest chance that a query lands in a cell, n ≥ log(K/δ)/π_min independent labels reach all K cells with probability at least 1 − δ (Appendix C.1), and this inverse-mass dependence cannot be avoided (Appendix C.2). The parser itself must be verified, and we have not measured the cost of model checking it. When several actions can change the safety state, each state needs a maximization over the logit box (Appendix B.3).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

General response to all reviewers

All three reviews rated the presentation 2, so this revision concentrates on clarity. "l.N" is a line number in the submitted PDF, and Appendices E–I are new.

Clarity and presentation. Besides the changes in the individual replies, Section 2 now defines supervision, coverage and a certificate (an upper bound on failure probability for every policy in a declared class). Sections 2–5 open by saying what they show, and Section 5 states the question each experiment isolates. Throughout, we split long sentences, reduce synonyms, and define terms such as history, router, gate, cell and critical state. The captions of Tables 1–3 and Figures 1–2 now state the setting, the units and the main finding.

Two corrections of our own. The introduction (l.40–43) gave a range of 0.06–0.17 nats for the student's KL divergence from the base at unselected histories, citing 1,728 policies. That range comes from only the 1,440 policies that have unselected histories, and the summary combined policies trained with and without rehearsal, a penalty that keeps the student near the base. The introduction now reports rehearsal weights 0.03 and 0.3 separately (Appendix D.3). Section 5.1 also notes that both prompt frequencies reuse the same seeds, so several runs in Table 1 share networks across frequencies.

What we do not claim. We do not claim guarantees for pretrained language models, that a learned gate can be trusted (l.47–48), or a lower bound on the number of queries a neural learner needs (l.469–476).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.