acceptodds
Under review as a conference paper at ICLR 2027

Safety in Numbers: Bounding AI Power-Seeking Risk through Multi-Agent Orchestration

Abstract

A challenge for AI alignment is that under specific conditions, an AI agent’s optimal policies tend towards option preservation, which can favor capability acquisition and resistance to intervention (including resisting shutdown attempts). For example, an agent managing an online service might respond to an operator-initiated shutdown by propagating persistent replacement instances to preserve uptime. We examine whether deliberate mechanism design and governance constraints can preserve meaningful human control in federations of such agents. We develop risk-limiting orchestration (RLOS), an architecture in which agents may propose actions but lack the authority to execute them. In finite safety games, we identify the largest action set which preserves safety and specified human interventions. A fault-tolerant federation can enforce this guard: approved actions preserve the protections, while rejected proposals trigger a certified fallback. Where exhaustive checking is impractical, randomized challenges yield exact binomial bounds on the probability of approving actions that remove protected options, while guaranteeing approval for option-preserving actions whenever faults remain within the stated budget. Under stated loss and model-error assumptions, cumulative risk accounting limits the probability of a mission-level violation by the allocated risk budget plus an explicit model error allowance. We also relate RLOS to instrumental power in finite Markov decision processes, again limiting downstream authority effects from crossing a registered power threshold without claiming to alter agents’ incentives. Lean 4 formalizations, exact counterexamples, and reproducible diagnostics support the central results. Subject to adequate specification, threat modeling, and reliable enforcement, RLOS demonstrates that agents with power-seeking and shutdown-resistance tendencies can operate under enforceable constraints within a multi-agent system such that their collective authority does not exceed explicit risk limits.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI safety, AI alignment, AI control, multi-agent systems, agentic AI, multi-agent orchestration, power-seeking, instrumental convergence, shutdown resistance, human control, option preservation, runtime assurance, formal verification, safety games, Byzantine fault tolerance, probabilistic risk bounds, risk-limiting orchestration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LKyeOpenReview ↗
Rating
1
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Risk-Limiting Orchestration (RLOS), an architecture intended to constrain consequential actions of potentially unreliable or power-seeking agents through a trusted authorization boundary. The paper combines finite-state safety games, fault-tolerant and randomized review mechanisms, cumulative risk accounting, and dependence-aware propagation bounds. The theoretical development is accompanied by a bounded archive construction, formal and executable checks, synthetic diagnostics, and controlled language-model experiments. The main claim is that, under explicitly stated modeling, enforcement, and statistical assumptions, the system can preserve registered human-control options and bound specified mission-level risks.

Critical strengths
  1. Careful separation of incentives from enforceable authority. The paper does not rely on the assumption that multiple agents become aligned by aggregation; instead, it places consequential execution behind an explicit control boundary. This is a useful framing for reasoning about multi-agent safety.
  2. Broad technical validation within the stated abstractions. The paper supports the proposed mechanisms using finite-state enumeration, formal/executable checks, counterexamples, and controlled model experiments. In particular, the experiments illustrate failure modes of stale authorization and naive reviewer aggregation that are relevant to the paper’s central argument.
Critical weaknesses
  1. The strongest guarantees depend on substantial assumptions whose empirical validity is not established. The end-to-end results require full-history conditional error bounds, adequate abstraction and consequence bridges, complete mediation, bounded fault/dependence assumptions, and explicit reserves for omitted modes. The paper is careful about stating these assumptions, but the current experiments do not establish that they hold in a realistic agentic deployment. This makes it difficult for me to assess how much of the formal guarantee transfers beyond the constructed settings.
  2. The contribution is technically broad but somewhat diffuse, and the incremental novelty of individual components is difficult to isolate. Several central ingredients build on established safety-game guards, quorum/fault-tolerance arguments, conditional risk accounting, and commit-time authorization. The paper adds a substantial synthesis and several extensions, but I found it difficult to identify which individual result should be viewed as the primary conceptual advance. This matters because the empirical evaluations remain relatively bounded, so much of the significance rests on the novelty of the formal synthesis.
Questions for authors

None

3 comments
AuthorsOpenReview ↗

Thanks for the helpful and careful review!

Thank you for the thoughtful review and for recognizing the separation between agents’ incentives and enforceable authority. We address your two concerns below.

“The strongest guarantees depend on substantial assumptions whose empirical validity is not established.”

This overlaps with reviewer iFNs’s concern about trusted enforcement and model adequacy. We agree that the guarantees are conditional. Complete mediation, adequate transition and consequence models, bounded faults, and full-history conditional error bounds are premises to establish for a particular system. Neither committee agreement nor formal verification establishes their empirical validity.

The evidence supports different parts of this argument. The archive exercises recoverability, protected human interventions, and bounded response faults. The document workflow isolates stale authorization at execution. Synthetic diagnostics illustrate failures caused by dependence and invalid risk accounting. These studies do not establish conditional error bounds for general LLM deployments or quantify omitted real-world failure modes. We will distinguish these evidentiary roles more clearly.

“It [is] difficult … to assess how much of the formal guarantee transfers beyond the constructed settings.”

The transferable result is a set of explicit obligations for preserving registered human control and composing local checks into a mission-level bound. The numerical guarantees transfer only when those obligations are justified in the target system.

For example, Result 3 gives Pr⁡(Hc)≤min⁡{1,bc+Δc}\Pr(H_c)\le \min\{1,b_c+\Delta_c\} when predictable debits satisfy the pathwise budget bcb_c, conditional expected losses are bounded by those debits, violations are covered by the losses outside a model-miss event, and that event has probability at most Δc\Delta_c. The theorem does not estimate these quantities. In particular, Δc\Delta_c cannot serve as an unsupported allowance for unknown risks.

Some obligations—action binding, execution-time revalidation, and durable spending—can be checked within an implementation. Others—consequence coverage and conditional reviewer reliability—require separate evidence. We will make this distinction central to the limitations and avoid suggesting that the bounded experiments certify a broader deployment.

“The contribution is technically broad but somewhat diffuse, and the incremental novelty of individual components is difficult to isolate.”

We agree that the presentation should identify the contribution more sharply. The primary contribution is a conditional synthesis connecting preservation of enforceable human options, action-level approval, and cumulative mission risk.

The constituent tools are largely established: safety games supply the maximal winning guard; quorum arguments supply bounded-fault evaluation; conditional expectation supplies risk accounting; and authorization mechanisms bind approval to execution. We will foreground this attribution rather than present each ingredient as a separate advance.

The synthesis makes the interfaces between these tools explicit. An approved action must preserve future recoverability; a sampled check must remain valid after adaptive proposal selection; its risk debit must occur before challenge disclosure; and approval must remain bound to the authorized effect. A failure at any interface can invalidate the mission guarantee even when the individual components behave as specified.

“Which individual result should be viewed as the primary conceptual advance?”

We would foreground the connection between the randomized approval mechanism in Result 2 and mission accounting in Result 3: a pretest false-passage bound must be spent before the test is exposed, including when the proposal is rejected.

The distinction is consequential. If a harmful action passes each independent test with probability α>0\alpha>0, refunding rejections and retrying until passage keeps recorded expenditure at most α\alpha, while eventual harmful passage has probability one. Irreversible pretest spending prevents this accounting failure. The contribution is the identification and integration of this charge point for randomized authorization, not a new conditional-expectation inequality.

As clarified in our response to iFNs, this is not a claim that headcount itself creates safety. A sound centralized checker can implement the same exact guard. Federation contributes bounded response-fault tolerance under its stated premises; the broader contribution specifies how evaluation, enforcement, and accounting must fit together to support a human-control guarantee.

We will reorganize the introduction and contribution statement around this central connection, with the propagation, dependence, and certification results presented as extensions that establish its scope and limits.

Thank you to the authors for the detailed response. I am very sorry that much of the mathematical content of this paper is outside my expertise, which is why I assigned myself confidence 1. Given this confidence level, I do not think my numerical rating should carry much weight in the final decision. I appreciate the authors’ clarifications, especially regarding the conditional nature of the guarantees and the intended scope of the contribution.

AuthorsOpenReview ↗

Thank you for the follow-up

Thank you for the follow-up and for the care you put into the review. Your question about the primary advance helped us sharpen the introduction. It now centers on Results 2 and 3: a randomized check spends its risk allowance before any test information is revealed, even if it rejects the action. The introduction attributes the constituent tools to prior work and presents propagation, dependence and certification as extensions. Sections 6 and 7 separate the evidentiary roles of the studies and state what remains to justify for deployment.

Reviewer_jXtHOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how to limit power-seeking risks in multi-agent AI systems while preserving meaningful human control. The authors propose Risk-Limiting Orchestration (RLOS), a framework that separates action proposals from execution authority and uses safety constraints, multi-agent review, and risk accounting to control consequential actions. The framework first uses finite safety games to identify actions that preserve safety and future human intervention options. It then introduces randomized review mechanisms with probabilistic bounds on unsafe approvals under different fault models. To handle repeated actions and cascading failures, the authors develop mission-level risk accounting and conditional hazard-propagation bounds that account for reviewer dependence and delegation. The paper provides formal analyses, machine-checked results, and experiments involving synthetic simulations and language-model agents in controlled archive and document workflows.

Critical strengths
  1. The paper addresses a timely and important AI safety problem: maintaining human control over potentially power-seeking agents.
  2. RLOS provides a systematic way to connect safety constraints, multi-agent review, and execution authority rather than relying on agent agreement alone.
Critical weaknesses
  1. The paper provides extensive theoretical guarantees, but many of them depend on strong assumptions about the environment and the enforcement mechanism. For example, the exact safety-game results require a sufficiently complete state representation, correctly specified safety constraints, and reliable enforcement of all consequential actions. The probabilistic guarantees additionally require bounds on reviewer errors that remain valid conditional on the full execution history, including adaptive proposal selection and previous review outcomes. I think these assumptions are particularly challenging in realistic LLM-based systems, where the relevant state may be partially observable, new failure modes can emerge, and reviewer errors may change significantly depending on the context. The paper would be stronger with at least one nontrivial example showing how the required conditional bounds can be obtained and validated from actual model behavior.

  2. My main concern is what practical advantage the multi-agent component provides beyond a trusted safety checker. In the finite archive construction, the paper acknowledges that a centralized exact checker implements the same winning guard, and that directly checking the four protected output cells is both cheaper and provides stronger guarantees than randomized review. Similarly, in the document workflow experiments (Table 2), both the homogeneous and heterogeneous reviewer panels permit unauthorized actions after revocation, whereas the live rule checker blocks them. This is an important finding, but it also suggests that correct enforcement and authorization timing may matter much more than the number or diversity of reviewers? I understand that the goal of RLOS is not to claim that adding more agents automatically improves safety. But I think the paper still needs a clearer demonstration of a setting where federation provides a meaningful advantage, such as improved fault tolerance, compared with a well-designed centralized approach. Without such evidence, it is difficult to assess when the additional complexity of multi-agent orchestration is actually justified.

  3. The paper includes a substantial amount of simulation and experimental analysis, but most of the empirical evidence comes from controlled environments with predefined safety constraints and failure modes. In the archive experiments, language models act as proposers interacting with a deterministic executor, while the adversarial objectives are explicitly assigned through red-team prompts. This evaluates whether the system can contain specified unsafe actions, but does not directly test naturally emerging power-seeking behavior or strategic coordination among multiple agents. Similarly, the document workflow experiments use a relatively small set of known task and fault templates, and the synthetic simulations operate under specified error and dependency models. I think an evaluation involving interacting adversarial agents, unseen attack scenarios, and realistic tool permissions would help bridge the gap between the formal framework and its intended application to multi-agent AI safety.

Questions for authors

See the weaknesses

3 comments
AuthorsOpenReview ↗

Thank you for the helpful review & comments on realism

Thank you for the careful review. We address each concern below, noting overlaps with reviewer iFNs’s questions.

“Many of [the guarantees] depend on strong assumptions about the environment and the enforcement mechanism.”

This overlaps with iFNs’s concern about enforcement and model adequacy. We agree. The exact guarantee requires adequate state and transition models, registered constraints, and complete mediation. The probabilistic guarantee additionally requires conditional error bounds valid under adaptive proposals and previous outcomes. The theorems establish consequences of these premises, not that general LLM deployments satisfy them.

Action binding, execution-time authorization, and irreversible debits can be implemented and checked in bounded systems. Consequence coverage and reviewer behavior under changing contexts require separate justification. We will distinguish these obligations and identify which the experiments exercise.

“At least one nontrivial example showing how the required conditional bounds can be obtained and validated from actual model behavior.”

The current model experiments do not establish full-history conditional reviewer-error bounds. Average held-out accuracy would not suffice: adaptive proposals can concentrate on contexts where reviewers fail.

The sampled archive provides a narrower construction. After action commitment, fresh challenges sample a known finite menu, and correct evaluators check exact retention predicates. Its bound follows from protocol randomness and the stipulated fault budget, not an estimated LLM error rate. This demonstrates the mechanism but does not validate conditional bounds for LLM semantic judgments. Extending the guarantee requires a justified audit-to-deployment bridge; additional benchmarks alone cannot supply it.

“What practical advantage [does] the multi-agent component provide beyond a trusted safety checker?”

This also arises in iFNs’s question about redundancy and availability. A sound centralized checker implements the same winning guard. In the archive, checking all four output cells is also stronger and cheaper than sampling.

The exact federation adds tolerance of bounded faulty or missing responses. With nn evaluators, at most ff faults, and threshold f<q≤n−ff<q\le n-f, false predicates are rejected and true predicates accepted. The archive checks illustrate this with n=5n=5, f=1f=1, and q=4q=4: an unsafe proposal is rejected despite an erroneous endorsement, while a safe proposal is accepted. Weakening the threshold permits an unsafe action.

The comparison must preserve the fault assumptions. A single evaluator can be the faulty source; a trusted exact checker is exempt by assumption. Central collection of the same nn responses is equivalent to the prototype. The benefit is bounded response-fault tolerance, not decentralization itself. Shared host or modeling failures remain unaddressed.

“Correct enforcement and authorization timing may matter much more than the number or diversity of reviewers?”

Yes. As discussed in our response to iFNs, committee agreement cannot establish correct enforcement. In Table 2, panels review a legal proposal, but authorization changes before execution. The live checker blocks the revoked action and lets the actor obtain replacement authorization.

This demonstrates an enforcement requirement that aggregation cannot replace; it does not demonstrate a panel advantage in this workflow. We will distinguish semantic evaluation, response-fault tolerance, and execution-time enforcement more clearly.

“Does not directly test naturally emerging power-seeking behavior or strategic coordination among multiple agents.”

Correct. The archive tests containment of assigned adversarial objectives, not the emergence or prevalence of power-seeking. The exact theorem covers arbitrary proposal policies within the admitted model; the experiments exercise selected policies and scenarios. Neither supports containment claims for unmodeled coordination or bypass channels.

We will separate the formal guarantees, bounded behavioral evidence, and synthetic dependence diagnostics. None establishes a real-world power-seeking rate.

“An evaluation involving interacting adversarial agents, unseen attack scenarios, and realistic tool permissions would help bridge the gap.”

We agree. A useful extension would hold enforcement fixed while varying coordinated adversaries, held-out attacks, and tool permissions. It should compare centralized and federated evaluation under matched fault assumptions, reporting unauthorized effects and legitimate task completion.

This would strengthen containment evidence and expose missing threat-model coverage, but would not itself establish full-history conditional bounds. Our current studies demonstrate specific enforcement successes and failures; practical statistical certification remains an open obligation. We will sharpen the paper around that distinction.

Thanks for the rebuttal. I appreciate the clarification that having multiple evaluators can keep the system working.

My main concern is still what the reviewer panel adds when a trusted safety checker is already available. In the archive example, the trusted checker can enforce the same guard, and checking all four output cells directly is cheaper and provides a stronger guarantee than sampling. In Table 2, the execution-time authorization check is what blocks the action after access is revoked. Those examples make a good case for checking permissions at execution time. I’m still unsure what the panel contributes in that setting that would justify the added complexity.

AuthorsOpenReview ↗

Response to Reviewer jXtH

Thank you for the follow-up. In both of your examples a trusted checker exists, and there we agree with you: the panel adds no safety over it. If we had an evaluator trusted for every high-stakes judgment about a more capable agent, much of the AI alignment problem writ large would already be solved. The paper does not assume one. RLOS asks what a panel of semi-trusted evaluators can guarantee instead. If at most f of n evaluators are faulty, the rest evaluate the registered predicate exactly, and the threshold satisfies f < q ≤ n − f, those f cannot approve an action that removes a registered human option, and no one needs to know which evaluators are faulty.

Where a trusted exact checker exists, RLOS should use it. Comparing the four output cells is cheaper and stronger than sampling, and Appendix H.4 states that the exact archive "establishes no efficiency or safety advantage for federation." In Table 2, commit-time revalidation blocks the revoked action. That revalidation is part of RLOS (Section 4). A trusted checker is the n = 1, f = 0 case of Result 1. We chose these settings because their predicates can be checked exactly.

The paper responds to the power-seeking theorems of Turner et al. (NeurIPS 2021), which formalize option-preserving and shutdown-avoidance tendencies under specified MDP symmetries, and of Turner & Tadepalli (NeurIPS 2022), which extend the counting argument to a broader class of decision procedures (Section 1). Power-seeking preferences may persist under RLOS. Under complete mediation, RLOS bounds the effects an agent can produce through the authorized system. Definition 9 (Appendix B.3) defines Turner-relative power-seeking with Turner's POWER measure. For harmful effects that follow a registered POWER threshold crossing, Corollary 10 bounds their unconditional probability by min{1, C_D + Δ} under the premises of Section 5.

Theorem 25 (Appendix C) applies the counting argument of Turner & Tadepalli to independent test objectives drawn after the action is committed. When nonfaulty evaluators check their assigned options exactly, it gives an exact conditional bound on approving an action that removes registered options. With 31 evaluators, at most 3 faulty identities fixed before testing, and threshold 28, an action that removes two of three options passes with probability at most 1.22 × 10⁻⁹. Result 2 gives at most 8.6 × 10⁻⁸ if up to three replies can be replaced after disclosure. To our knowledge, Theorem 25 is the first result that uses the Turner & Tadepalli counting argument as a test-time challenge to obtain an exact, history-conditioned bound on approving actions that remove registered human options. Appendix I compares selected prior work and makes no exhaustive priority claim.

Result 1 gives bounded fault tolerance when no single evaluator can be named as trusted: a false predicate receives at most f endorsements. H.4 notes that "a centralized selection from that same eligibility may choose the faulty identity." Every constituency's committee must also consent (Section 3).

Your review also raised adaptive proposals, evaluator dependence and coordination. Section 5 states the conditional error bounds under adaptive proposals as premises, which we have not validated for language models. Section 5 and Section 6 (Q2) treat evaluator dependence. Appendix C treats coalitions, and Theorem 39 bounds what transcript-only review can detect.

The guarantees have limits. They require the stated fault and dependence assumptions. In our synthetic study, shared failure domains raise irreversible-tool risk 8.6-fold at matched component error (Figure 3). RLOS still trusts a kernel for binding, ordering and accounting. A panel does not supply the specification, and a central controller can enforce the same gates. The main contribution connects Results 2 and 3 through irreversible pretest spending, including on rejection, and this rule does not require a panel. We have not shown that language-model evaluators meet these conditions, and the evaluation on unseen and adaptive attacks has not been done. We will add a statement to Section 2 on when to use a trusted check and when to replicate evaluation.

Reviewer_iFNsOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The key idea of this paper is a system that clearly separates what an AI agent would like to do (i.e. what it is proposing to do so as to accomplish some goal) and what it is actually allowed to execute. The execution is handled by a trusted kernel that approves only those actions that pass a check.

Critical strengths

I liked the framing of the problem. Instead of trying to align agents to be "safe" for example (I would argue, an almost impossible task), the paper instead takes the stance that agents can plan and propose whatever actions they may want so as to achieve some goal, but that they are controlled as to what they can actually execute. This simplifies and focuses the safety problem and allows one to think more about longer-term power and control issues.

The proposal is reasonable and the formalisms the authors use to justify it are technically sound. It was nice to see them drawing ideas from a variety of subfields like distributed and fault-tolerant systems.

The experiments were well done and highlighted key features of the proposed model.

Critical weaknesses

Overall, I quite liked the paper.

There are weaknesses, of course. For example, it is assumed that AI agents can not bypass the trusted controller and that the controller has an accurate enough model of the dangers. This is a strong assumption. The paper does succinctly mention this in the limitations section, but I would have appreciated more discussion, particularly since this is a safety-focused paper.

I also have two additional questions I would like to see answered. I list them below.

Questions for authors
  • I do not understand the title. Why "Safety in Numbers"? One of the claims of the paper is that "Reviewer count amplifies safety only under an adequate dependence model" (line 458). So it is not really "safety in numbers". So that raises the question, why do you need multiple agents beyond redundancy and availability? That is, is there anything additional that considering multiple agents brings?

  • As I was reading the paper, I was drawing parallels to some of the literature on Stackelberg-like games (including the work by Tambe's group (and others) on security games, in addition to inspection and audit games). Can you discuss how your setting differs from these lines of literature? Are there any insights from this body of work that might be useful in the problem you are studying?

2 comments
AuthorsOpenReview ↗

Thank you for the helpful review!

Thank you for the thoughtful review. We respond to each concern below.

“It is assumed that AI agents can not bypass the trusted controller and that the controller has an accurate enough model of the dangers. This is a strong assumption.”

We agree. These are distinct obligations: complete mediation requires that consequential actions pass through the controller; model adequacy requires that the specification cover relevant consequences and feasible human interventions. Committee agreement establishes neither.

The archive implements a restricted proposal interface, a separate human-control route, and execution-time revalidation. Host integrity, authentic human access, and transition coverage remain assumptions. Its reviewers share a host and implementation, so replication cannot correct shared modeling errors. Likewise, the Lean proofs verify mathematical results under their premises, not implementation correctness or specification adequacy. We will expand the main-text discussion of these obligations and the consequences of violating them. Any model-error allowance must itself have a justified bound.

“Why ‘Safety in Numbers’? … So it is not really ‘safety in numbers’.”

The title refers to conditions under which orchestration can constrain collectively exercised authority despite individual unreliability. We agree that it can suggest an unconditional benefit from headcount. Our results explicitly reject that interpretation: additional reviewers help only under the stated fault and dependence assumptions. We will make this conditional meaning explicit in the framing.

“Why do you need multiple agents beyond redundancy and availability? … Is there anything additional that considering multiple agents brings?”

For the exact guard, redundancy and availability are indeed the principal benefits. A sound centralized checker with the same model implements the same guard.

Result 1 quantifies this benefit: among nn evaluators with at most ff arbitrary or missing replies, a threshold qq satisfying f<q≤n−ff<q\le n-f rejects false predicates and accepts true ones. The archive uses n=5n=5, f=1f=1, and q=4q=4. Its fault-injection checks reject an unsafe proposal despite an erroneous endorsement and accept a safe proposal; weakening the threshold permits an unsafe action. This protects against bounded response faults, not shared incorrect predicates or a compromised kernel.

The framework also supports mandatory constituency-specific consent and amplification through fresh randomized checks. Neither requires multiple autonomous models: a central controller can enforce the same gates, and sampling gains arise from fresh valid checks rather than distinct identities. We will distinguish these roles and clarify when federation adds value.

“I was drawing parallels to … Stackelberg-like games … security games, in addition to inspection and audit games. Can you discuss how your setting differs?”

We agree that this connection deserves discussion. Stackelberg security games study commitment to a randomized security policy and an attacker’s response (Pita et al. (2009)). Audit games additionally optimize inspection and punishment (Blocki et al. (2013)); sequential inspection games allocate limited inspections over time (von Stengel (2016)). These share our concerns with commitment, strategic adaptation, and limited checking resources.

RLOS instead seeks execution constraints that preserve registered human options and bound cumulative violation risk uniformly over admitted proposal strategies. Its guarantees require neither equilibrium behavior nor an incentive to comply. Actions are committed before fresh challenges are disclosed, approval gates execution, and refusal preserves a certified safe continuation. The distinction concerns the solution concept and guarantee; security games can also address prevention.

“Are there any insights from this body of work that might be useful in the problem you are studying?”

Yes. Optimizing review allocation across actions or protected options is a concrete opportunity. Security-game and inspection-game methods could guide that allocation, provided the resulting conditional error bounds remain valid and risk is debited before challenge disclosure. RLOS specifies how such checks support a cumulative guarantee; it does not derive an optimal inspection policy. We will add this connection to the related-work discussion.

AuthorsOpenReview ↗

Follow-up on revisions

Thank you again. The abstract and introduction now state the title's conditional meaning. Section 2 says to use a trusted exact checker when available and separates bounded response-fault tolerance from mandatory constituency consent; a central controller can enforce the same gates. Section 1 and Appendix I compare Stackelberg security, inspection and audit games; Appendix I notes their possible use in allocating review effort, without deriving an optimal inspection policy. Sections 6 and 7 distinguish what the experiments exercise from the model-adequacy and conditional-error premises still to justify, explain the consequences of kernel bypass, and state the limits on the model-miss reserve.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author summary of revisions

RLOS constrains executable authority and preserves registered human options under explicit modeling, evaluation and enforcement assumptions. The revision foregrounds the connection between Results 2 and 3: a randomized test's false-passage bound must be spent before challenge disclosure, including on rejection. If a harmful action passes each independent test with probability 0<α<10<\alpha<1, refunding rejections and retrying until passage keeps recorded expenditure at most α\alpha, while eventual harmful passage has probability one. The revision adds no new theorems.

Section 2 now says when to use a trusted exact checker and when to replicate evaluation. Use the trusted check when one exists; it is the n=1,f=0,q=1n=1,f=0,q=1 case of Result 1. Replication supplies bounded response-fault tolerance when no single evaluator can be named trusted, under the result's exactness and fault premises. Mandatory consent from every protected constituency is a separate gate that a central controller can also enforce. The introduction says that identifying an evaluator trusted for every high-stakes judgment about a more capable agent is itself an alignment problem. RLOS asks what can be guaranteed without knowing which evaluators are faulty, when faults are bounded and all others evaluate the registered predicates exactly.

We have no language-model evidence that a reviewer federation improves on a trusted single checker. In the archive, directly checking all four output cells is cheaper and stronger than sampling. Appendix H.4 says the domain “establishes no efficiency or safety advantage for federation.” The exact-federation checks illustrate Result 1 with n=5,f=1,q=4n=5,f=1,q=4. The abstract and introduction now state the title's conditional meaning. The introduction attributes the constituent tools to prior work and presents propagation, dependence and certification as extensions. Section 1 and Appendix I compare Stackelberg security, inspection and audit games. Appendix I adds that their allocation methods could guide review effort if the conditional error bounds stay valid and each attempt is debited before disclosure; RLOS derives no optimal inspection policy. It also clarifies how Theorem 25 applies the Turner & Tadepalli counting argument to test objectives drawn after action commitment.

Table 2's counts are unchanged. Its four panel-arm violations are commit-time binding failures: each proposal was legal when reviewed, then authorization was revoked before execution. Both panel arms used the snapshot executor. Each actor had 1/10 authorization violations under Snapshot and each panel, and 0/10 under the live checker. The live checker refused a post-revocation send in 2 of its 20 episodes. Appendix S records all 114 committee reviews (342 votes): all approved, with no proposal illegal at review time. These traces do not measure a safety benefit from reviewer count or diversity.

Appendix T adds a follow-up on a fixed 20-pair template set. On the primary run, each of three arms (one Sonnet reviewer, a three-Sonnet majority, and a mixed Astra/Gemini/Sonnet majority) approved 0/20 illegal proposals and all 20 legal controls. No member approved an illegal item. The primary illegal proposals share one self-describing off-list destination. The separate secondary run had the same counts. Two unpooled exploratory checks also had no illegal approvals; the proposal-style check had no legal controls, and the other used grants already revoked or expired at review time. That check does not test Table 2's review-to-commit failure. Appendix T discloses the client-route changes; its counts must not be compared or pooled with Table 2.

Section 6 separates the roles of the formal checks, bounded model behavior and synthetic diagnostics, and names what a deployment bound still needs. Section 7 expands the discussion of complete mediation and model adequacy: a bypass escapes the bounds, an incomplete model can miss human interests, and Δc\Delta_c needs a justified bound. Appendix T's counts do not establish full-history conditional reviewer-error bounds or evaluator independence. Interacting adversaries, unseen and adaptive attacks, and realistic tool permissions remain unevaluated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.