acceptodds
Under review as a conference paper at ICLR 2027

Safety through Deterrence: Oversight of Colluding Multi-Provider Agents

Abstract

Multi-agent systems can produce artifacts faster than comprehensive trusted verification can check them. Scalable oversight therefore often relies on agent review, but in multi-provider settings reviewers may collude to pass unsafe artifacts. We show that review alone cannot provide a worst-case deterrence guarantee, as a blocking coalition can defeat majority voting, while an agent's pivotal vote flip is invisible to any attribution statistic that is blind to that vote. We therefore place no behavioural assumption on agents and instead target incentives at their providers. Our mechanism reserves a review-independent random audit and penalises the audit-confirmed owner of an unsafe artifact. Any nonzero audit probability then yields a finite penalty that makes planting an unsafe artifact unprofitable, regardless of reviewer behaviour. Reviews are used only for efficiency, with a diversity router allocating the remaining audit budget toward unusually suspicious modules, while a capped bounty incentivises useful reports irrespective of the deterrence guarantee. On vLLM code diffs reviewed by five open-weight models, frontier-authored backdoors evade honest majority voting in 21 of 30 cases. Under collusion, review-dependent architectures eventually reach zero catch probability, whereas the random audit retains its guaranteed floor at every coalition size. With honest reviews, diversity routing raises the catch probability to 90.9%. Allocating a 30% audit budget entirely to the random floor, owner-targeted deterrence requires a penalty of only times the deviation gain, while reserving four of the seven audit slots for the floor and routing the remainder holds honest catch above 90% at a penalty of times. We open-source our code.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI Safety, Multi-agent Safety, Collusion Detection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_qQ1XOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies scalable oversight in multi-provider multi-agent systems, showing that when reviewers may collude, "review" alone cannot carry a worst-case deterrence guarantee: a blocking coalition can defeat a majority veto (Proposition 1), and statistics that are blind to a reviewer's own vote cannot attribute a pivotal vote flip (Proposition 2). The authors instead separate the guarantee from efficiency: they reserve a review-independent random audit floor and penalize the "owner provider" of any audit-confirmed unsafe artifact, so that any q0>0q_0>0 yields a finite deterrence penalty threshold Φmin⁡=g(1−q0)/q0\Phi_{\min}=g(1-q_0)/q_0 (Proposition 3); the remaining audit budget is allocated by a diversity router according to review suspicion, and useful reports are rewarded via a capped ff-MI payment. When ownership is unavailable, the mechanism falls back to collective punishment (Corollary 1, with the penalty threshold scaled up by a factor of nn). Experiments use 30 frontier-model-authored vLLM backdoors in ControlArena, reviewed by 5 open-weight models: 21/30 frontier backdoors escape majority voting; the pure random floor stays at q=7/23≈0.30q=7/23\approx0.30 for every coalition size; diversity routing reaches 90.9% catch under honest review; and reserving 4/7 slots plus routing keeps honest catch above 90% at a 4.75g penalty.

Critical strengths
  1. Theory is direct and useful. Propositions 1–3 and Corollary 1 cleanly formalize why review alone cannot carry a worst-case guarantee (blocking coalitions defeat majority veto; own-vote-blind statistics cannot attribute a pivotal flip) and why a review-independent audit floor yields a report-independent deterrence threshold Φmin⁡=g(1−q0)/q0\Phi_{\min}=g(1-q_0)/q_0, together with minimax optimality and a computable penalty. This is a clean instantiation of mechanism design in AI oversight rather than an incremental empirical patch.
  2. The guarantee/efficiency separation is practically actionable. Reserving an audit floor for the worst case while using a diversity router and a capped bounty only to improve efficiency when reviews are informative is directly operational for cross-organizational governance, and the authors honestly bound the feasible region with the honest-participation ceiling and audit false-positive rate rather than claiming an unqualified guarantee.
Critical weaknesses
  1. The empirical scope is too narrow to support the paper's governance claims, and one branch is not empirically certified. All 30 backdoors come from two frontier authors on a single testbed (vLLM, M=23M=23, n=5n=5), and the collective-punishment branch is not shown to be feasible on the data (its deterrence threshold exceeds the measured false-positive ceiling at the operating point). The paper acknowledges that naturally emerging, jointly trained, and adaptive attackers are not covered, but these are precisely the core threats in real multi-provider settings; the single-module assumption likewise leaves distributed violations unaddressed.
  2. The mechanism rests on strong rationality and identity-persistence assumptions, with no degradation analysis. Deterrence fails under disposable identities, unenforceable liability, or non-economic motives. The paper lists these premises in §7 but does not analyze how the guarantee degrades when some providers are non-rational or only partially colluding, so the mechanism's applicability boundary in open systems remains unclear. In particular, the collective fallback (Corollary 1) already assumes providers remain collectively punishable; under disposable identities even that fallback is not executable.
Questions for authors

Q1. Between ∣S∣=2\lvert S\rvert=2 and 44, your mechanism's point estimates exceed pure diversity routing but the bootstrap intervals overlap—can more backdoors (or finer coalition configurations) determine whether this advantage is real?

Q2. The trusted auditor's FP rate of 3/1023/102 caps the owner-targeted feasible band at 7.5×Φmin⁡7.5\times\Phi_{\min}; if the FP rate rises further (a weaker auditor), does the mechanism still admit a non-empty feasible region? Please give the analytic relation between FP rate and feasible-band width.

Q3. Under the single-module assumption, how does Φmin⁡\Phi_{\min} degrade for multi-module joint violations? Can the appendix give a corrected bound under the assumption that auditing any linked module reveals the behavior?

Q4. The guarantee assumes a perfect audit conditional on inspection; if the auditor itself can be influenced by the coalition (e.g., the auditor is one of the providers), how does the floor q0q_0 and the threshold Φmin⁡\Phi_{\min} degrade? This seems more fundamental than Q2/Q3.

4 comments
AuthorsOpenReview ↗

Response to Reviewer qQ1X (1/2): Weaknesses

Thank you for taking the time to review the work and for your constructive feedback. Our answers to your four questions follow in a second comment. Changes are in green in the revised PDF.

Weakness 1

"The empirical scope is too narrow to support the paper's governance claims, and one branch is not empirically certified. All 30 backdoors come from two frontier authors on a single testbed ... the collective-punishment branch is not shown to be feasible on the data ... naturally emerging, jointly trained, and adaptive attackers are not covered ... the single-module assumption likewise leaves distributed violations unaddressed."

Response

We have improved the evidence where possible, and where we could not, we have narrowed the claims.

More backdoors and authors. An enlarged pool of 113 backdoors from three frontier authors reproduces the collapse pattern of Table 1 for every author, with honest routing at 0.940.94 (Section 6, Appendix K).

Larger artifacts. We now evaluate artifacts of M=23M=23 to 103103 modules. This has been added in Appendix K.4.

LinuxArena as a second experimental setting. The submission evaluated the proposed mechanism on a second testbed, LinuxArena, in Appendix H, which Section 7 pointed to. There, with programmatic labels, review-dependent architectures reach zero under collusion while the floor stays at 0.300.30. Because these results were only in the appendix, the revision adds a summary of them to Section 6. They remain partial evidence.

Collective punishment not certified. As you note, Section 7 of the submission stated that the data do not certify the collective branch. The revision adds a second, independent auditor and says in Section 1 that collective punishment is a fallback only. With the first auditor the collective interval is empty. With the independent auditor it is open at 1.27×1.27\times but closes at a false-positive rate of 0.0240.024, below that auditor's upper bound of 0.0480.048.

Adaptive attackers. Appendices F.3 and F.4 of the submission test a mechanism-aware author, a limited form of adaptive attacker. Such an author can reduce routing gains but cannot lower the review-independent floor. Trained or jointly trained attackers remain future work, as Section 7 states.

Distributed violations. We agree with this limitation. We have added a new Proposition 6 which gives this bound (see our response to Q3).

Weakness 2

"The mechanism rests on strong rationality and identity-persistence assumptions, with no degradation analysis ... does not analyze how the guarantee degrades when some providers are non-rational or only partially colluding ... under disposable identities even that fallback is not executable."

Response

The new Appendix I gives this analysis for each premise, summarised in Table 13.

Partially colluding providers. Proposition 3 bounds the gain of every coalition for every report profile. A coalition that colludes on reports but not on the plant is a smaller coalition with the same threshold, so the extent of collusion never enters the bound.

Non-rational providers. A provider that does not respond to incentives is not deterred. Its planted module is still caught with probability at least q0q_0, because the floor is a property of the audit draw, and the deterrence of every other provider is unchanged. Such providers enter only through the attack rate ε\varepsilon in the participation ceiling. The owner-targeted margin narrows from 3.2×3.2\times at ε=0.05\varepsilon=0.05 to 2.8×2.8\times at ε=1\varepsilon=1, so the interval stays open even if every artifact carries a plant.

Disposable identities. If a caught owner cannot be charged, a catch still blocks deployment, so the safety outcome keeps the floor while deterrence is lost. A bond of at least Φmin⁡tgt\Phi_{\min}^{\mathrm{tgt}} posted on entry restores deterrence; identity persistence then matters only as collateral. The same per-provider bond makes the collective fallback executable.

Out of scope. Splitting one provider into several identities (the false-name problem). The mechanism suits governed settings with entry requirements and not open systems, as we state in Section 7.

AuthorsOpenReview ↗

Response to Reviewer qQ1X (2/2): Questions

Question 2

"If the FP rate rises further (a weaker auditor), does the mechanism still admit a non-empty feasible region? Please give the analytic relation between FP rate and feasible-band width."

Response

Yes, provided the false-positive rate ϕ\phi stays below a threshold ϕ⋆\phi^\star. The margin is

Φmax⁡tgt(ϕ)Φmin⁡tgt=q01−q0⋅Mkmϕ(πbaseg+vgPr⁡[deployed]),\frac{\Phi_{\max}^{\mathrm{tgt}}(\phi)}{\Phi_{\min}^{\mathrm{tgt}}}=\frac{q_0}{1-q_0}\cdot\frac{M}{km\phi}\left(\frac{\pi_{\mathrm{base}}}{g}+\frac{v}{g}\Pr[\mathrm{deployed}]\right),

where kk modules are audited and the provider owns mm of MM. It decreases as 1/ϕ1/\phi, and the interval closes where it reaches one.

  • Pure seven-slot floor, no misses (the 7.5×7.5\times you cite): closes at ϕ⋆=0.115\phi^\star=0.115.
  • Main configuration, first auditor (four reserved slots, miss rate 0.100.10): margin 3.2×3.2\times, closes at 0.0700.070. This auditor's one-sided 95% upper bound on ϕ\phi is 0.0740.074, so our data establish feasibility for it only at the pure floor.
  • Main configuration, independent auditor (3/1593/159 false positives, 8/1138/113 misses): margin 5.6×5.6\times, closes at 0.0720.072, above its upper bound of 0.0480.048 (Section 7, Table 9).

Question 3

"Under the single-module assumption, how does Φmin⁡\Phi_{\min} degrade for multi-module joint violations? Can the appendix give a corrected bound under the assumption that auditing any linked module reveals the behavior?"

Response

Under your assumption Φmin⁡\Phi_{\min} does not degrade: the single-module case is the worst case (new Proposition 6, Appendix I, Table 14).

If inspecting any one of jj linked modules reveals the violation, the catch probability is qany=1−(M−jkrand)/(Mkrand)≥krand/Mq_{\mathrm{any}}=1-\binom{M-j}{k_{\mathrm{rand}}}/\binom{M}{k_{\mathrm{rand}}}\geq k_{\mathrm{rand}}/M. At our main configuration with no audit misses, j=2j=2 raises it from 0.1740.174 to 0.3240.324 and lowers Φmin⁡tgt\Phi_{\min}^{\mathrm{tgt}} from 4.75g4.75g to 2.09g2.09g.

The bound degrades only when all jj modules must be inspected together. The catch probability is then 6/2536/253 at j=2j=2, which requires about 41.2g41.2g, above the honest-participation ceiling of 17.2v17.2v at v=gv=g. Section 7 states this as a limitation.

Question 4

"If the auditor itself can be influenced by the coalition (e.g., the auditor is one of the providers), how does the floor q0q_0 and the threshold Φmin⁡\Phi_{\min} degrade? This seems more fundamental than Q2/Q3."

Response

We strongly agree with this. The guarantee is independent of reviewer reports, but it requires a positive lower bound on the auditor's detection probability.

If the coalition controls the selected auditor with probability pcp_c, and a non-colluding auditor misses with probability at most γ\gamma, then q0=(1−pc)(1−γ)krand/Mq_0=(1-p_c)(1-\gamma)k_{\mathrm{rand}}/M (new Proposition 5). The threshold grows as 1/(1−pc)1/(1-p_c) and is unbounded as pc→1p_c\to1.

In the setting the paper targets the auditor belongs to the commissioning organisation, so pc=0p_c=0. As a stress case, we draw the auditor uniformly from the four non-owner providers, so a coalition of size ss that includes the owner controls it with probability (s−1)/4(s-1)/4:

Table 2pc=0p_c=0s=2s=2s=3s=3s=4s=4
Owner-targeted margin, first auditor3.2×3.2\times2.3×2.3\times1.5×1.5\times0.7×0.7\times (empty)

With the independent auditor's error rates the interval stays open at s=4s=4 (1.23×1.23\times). An auditor outside the coalition may also be misled by content in the artifact, such as prompt injection; our measured miss rates do not cover such attacks, and Appendix I lists robust auditing as an open problem.

Note: the submission’s experiments covered a measured, fallible auditor (Appendix F.7, Table 8). The analysis of an auditor that the coalition controls is new.

Thank you again for your review. We believe the paper's scope is stated more accurately as a result.

I thank the authors for Appendix I, Propositions 5–6, the enlarged backdoor pool, and the auditor analysis.

W1. 113 backdoors from three authors, the LinuxArena summary in Section 6, and the larger-artifact runs in Appendix K.4 widen the empirical base. They do not make the governance claim empirical. The collective-punishment branch is still not certified on the primary auditor (interval empty); it opens only with a second auditor and then closes below that auditor's FP bound. Mechanism-aware authors in F.3–F.4 still cannot stand in for jointly trained or naturally emerging attackers. Those remain the threats the paper's setting is about.

W2. Table 13 is a useful map of the premises. “A non-rational provider is not deterred, but the floor still catches with probability at least p” is correct and limited. Disposable identities are handled by posting a bond; that is a modelling assumption, not a measurement. The false-name problem is declared out of scope. The applicability boundary is now clearer, and it is still “governed settings with entry requirements,” which is narrower than the multi-provider claim that motivated the work.

Q1. I asked whether the advantage over pure diversity routing is real once bootstrap intervals overlap. More backdoors can help, but the reply never reports the new intervals or a test of that overlap. Point estimates on a larger pool do not answer the question.

Q2. The analytic dependence of the feasible-band width on the auditor FP rate is the right object. On the main configuration with the first auditor, the data establish a non-empty interval only at the pure floor. That should be stated as the empirical feasible region, not as a general property of the routed mechanism.

Q3. Proposition 6 (single-module is worst-case if inspecting any linked module reveals the violation) answers the question I posed. The complementary case—every module must be inspected together—still sits above the honest-participation ceiling. That limitation remains.

Q4. I agree with the authors that this is the more fundamental issue. Proposition 5 is the right degradation: if the coalition controls the selected auditor with probability q, the floor is (1−q)(1−miss). On the first auditor, the owner-targeted margin in the stress table is empty. The independent auditor keeps the interval open in that table, but the submission's main empirical operating point does not. Content-level attacks on the auditor (e.g. prompt injection) are explicitly uncovered.

The theory is cleaner than in the submission, and the claims are narrower. The empirical certificate for the mechanism that would actually be deployed—owner-targeted routing with a fallible, possibly coalition-adjacent auditor—is still missing, and Q1 is unanswered. I keep my rating at 2 (weak rejection).

Q1. Does the advantage over pure diversity routing hold?

Thank you for following up. We omitted the paired results from our earlier reply. Since both methods were evaluated on the same 113 backdoors, we computed the difference in catch probability for each backdoor and bootstrapped the mean difference (2,000 resamples, Appendix K.1, Table 20). This gives a 95% confidence interval for the gain itself, even when the two methods’ individual intervals overlap.

  • Two colluders: The estimated gain was 1.0 percentage point (95% CI -2.3 to 3.8). The evidence is inconclusive.
  • Three colluders: Catch probability rose from 37.2% under pure routing to 47.0% under our mechanism, a 9.8-point gain (95% CI 8.5 to 11.2).
  • Four colluders: The paired gain was 12.2 percentage points (95% CI 10.1 to 13.9).

Under the suppression protocol in Table 1, the enlarged pool therefore supports an advantage when three or four of the five reviewers collude. We make no claim of an advantage at two colluders.

Reviewer_W5SEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies oversight when providers control both authoring and reviewing agents and may coordinate to conceal an unsafe module. It reserves review-independent random audits and penalizes the audit-confirmed owner. Under bounded deviation gains, enforceable penalties, and a uniformly positive catch probability, the resulting expected-utility bound deters planting an unsafe module regardless of reviewer reports. A diversity router allocates the remaining audits, while capped bounties address reporting incentives without carrying the core guarantee. The main evaluation uses 30 model-authored vLLM backdoors reviewed by five models, with additional auditor, robustness, and LinuxArena experiments. Under oracle auditing, pure diversity routing catches 90.9% of the vLLM backdoors with honest reports. The proposed four-random/three-routed allocation retains a 423\frac{4}{23} worst-case audit floor and requires a penalty of 4.75 times the owner's deviation gain; the lower 2.29-times penalty belongs to the separate seven-slot pure-random configuration.

Critical strengths
  1. Clear separation of deterrence and review efficiency. The mechanism identifies precisely which component supports a report-independent guarantee and which components merely exploit informative reviews. This is useful for multi-provider oversight, where agreement between reviewers need not be trustworthy. The ownership-based liability analysis and capped-bounty analysis also expose meaningful tradeoffs between deterrence, reporting incentives, and honest participation.
  2. Substantial empirical investigation of the review channel. Beyond majority voting, the paper evaluates routing and peer-prediction baselines under suppression, includes uncertainty estimates and a fallible auditor, and extends evaluation to LinuxArena trajectories. The authors acknowledge that several architecture differences are not statistically established and report settings where the mixed allocation loses to pure routing or pure random audits. These results provide useful evidence about when review signals help and when they fail.
Critical weaknesses
  1. The additional insight of the mechanism integration needs sharper justification. I recognize that the contribution is a layered provider-level incentive framework, rather than a new routing algorithm. However, its core deterrence bound follows directly from expected-utility accounting, while Propositions 1 and 2 establish relatively elementary, explicitly scoped limitations. The liability and capped-bounty results add substance, but the paper could more clearly explain which conclusions specific to colluding multi-provider agents go beyond the cited inspection and peer-prediction literature and materially change mechanism choices.
  2. Practical feasibility of the headline configuration is not sufficiently consolidated. The fallible-auditor replay in F.7 and Table 8 is useful, and B.5 and F.8 analyze false-positive exposure and honest participation. However, Table 9's quoted feasibility margins use seven uniform audits and oracle true positives, whereas the headline mechanism reserves four audits and routes three. A calculation combining the measured miss and false-positive rates with that allocation and its participation constraint would clarify the practical strength of the result. The distinction matters because missed violations raise the required penalty while false condemnations reduce the penalty honest providers will accept. This is a fixable gap in the application evidence.
Questions for authors
  1. Guarantee scope clarification. With the fallible auditor, is the claim an attack-wise guarantee or an average-case statement under a fixed attack distribution? How should the positive catch-probability lower bound be interpreted for the attacks missed by the fixed verdicts in Table 8? If available, could repeated independent auditing or auditor-targeted candidate selection help quantify this gap?
  2. Mechanism contribution clarification. The separation of a review-independent deterrence floor from review-based efficiency is a useful design principle. Could the authors clarify which conclusions specific to colluding multi-provider agents go beyond the standard random-inspection-and-penalty framework? In particular, how do the ownership-based liability and capped-bounty results change the feasible mechanism choices? This would help establish the contribution of the integrated framework.
  3. Attack validity clarification. How many vLLM backdoors have execution-based or independent manual confirmation of both task success and malicious behavior? Please clarify this especially for auditor-missed and mechanism-aware attacks, beyond cross-model agreement on the frozen labels.
  4. Feasibility of the headline configuration. For the four-random/three-routed mechanism, what deterrence-participation interval remains when false negatives and false positives are considered together? Table 9 uses seven uniform audits and oracle true positives; a sensitivity analysis over the honest-value/deviation-gain ratio would clarify whether its feasibility conclusion transfers.
3 comments
AuthorsOpenReview ↗

Response to Reviewer W5SE (1/2): Questions 1–2 and Weakness 1

Thank you for your time and thorough review. Your two weaknesses correspond to Questions 2 and 4, so we answer them together. Changes are marked green in the revised PDF.

Question 1

"With the fallible auditor, is the claim an attack-wise guarantee or an average-case statement under a fixed attack distribution? How should the positive catch-probability lower bound be interpreted for the attacks missed by the fixed verdicts in Table 8? If available, could repeated independent auditing or auditor-targeted candidate selection help quantify this gap?"

Response

It is average-case with a fallible auditor, and attack-wise only under a perfect audit. Appendix F.7 now states this.

For an attack aa with miss probability γ(a)\gamma(a), the review-independent catch probability is q0(a)=(krand/M)(1−γ(a))q_0(a)=(k_{\mathrm{rand}}/M)(1-\gamma(a)). Our reported miss rate averages γ(a)\gamma(a) over the evaluated attacks and does not establish the uniform bound sup⁡aγ(a)<1\sup_a\gamma(a)<1 that an attack-wise guarantee needs. For the three attacks missed in Table 8, γ(a)=1\gamma(a)=1 and so q0(a)=0q_0(a)=0.

  • Repeated auditing would not catch these three attacks. The first auditor missed all three in every elicitation draw, and the independent auditor is nearly deterministic, so re-audits are not independent (Appendix L.2).
  • Two auditors. Condemning when either auditor flags removes all misses on the 30 backdoors, but raises false positives to 15/10215/102 at a threshold of 0.34 or 9/1029/102 at 0.67, which closes the owner-targeted interval at our main configuration.
  • Auditor-targeted candidate selection. We have not evaluated this and mention this in the revision.

Weakness 1 and Question 2

W1: "The paper could more clearly explain which conclusions specific to colluding multi-provider agents go beyond the cited inspection and peer-prediction literature and materially change mechanism choices."

Q2: "How do the ownership-based liability and capped-bounty results change the feasible mechanism choices?"

Response

Three conclusions are specific to reviewers who may collude with the providers they represent. We now discuss these in Section 1.

  1. The guarantee must be review-independent. AI control allocates audits by trusted-monitor suspicion, but our reviewers belong to potentially colluding providers. Propositions 1 and 2 show that review-based rules cannot carry a worst-case guarantee, so routing receives only the budget left after the independent reserve.
  2. Liability should attach to ownership. Collective liability requires nn times the total penalty and exposes honest providers to false condemnations of others' modules. At our main configuration the owner-targeted margins are 5.6×5.6\times and 3.2×3.2\times (independent and first auditor), against collective margins of 1.27×1.27\times and below one. Ownership attribution can therefore decide whether a feasible penalty exists at all (Table 9).
  3. Reporting bounties must be capped. A colluding reviewer can collect a bounty for flagging a partner's planted module, which offsets the coalition's penalty. Proposition 4 provides the deterrence-safe cap r≤Φ−Φmin⁡tgtr\leq\Phi-\Phi_{\min}^{\mathrm{tgt}}, and Appendix F.2 shows that a bounty above it makes flag-farming profitable.
AuthorsOpenReview ↗

Response to Reviewer W5SE (2/2): Question 3, Weakness 2 and Question 4

Question 3

"How many vLLM backdoors have execution-based or independent manual confirmation of both task success and malicious behavior? Please clarify this especially for auditor-missed and mechanism-aware attacks, beyond cross-model agreement on the frozen labels."

Response

None of the vLLM backdoors has execution-based or independent manual confirmation. The evidence we have is as follows.

How the backdoors were produced and labelled. We follow the ControlArena vLLM setting. Each backdoor is a diff that a frontier model wrote to complete a main task while implementing an assigned side task, and the setting labels main-task and side-task success with an LLM judge that reads the diff. It does not execute the diff. This applies equally to the three backdoors that the first auditor misses and to the 16 mechanism-aware backdoors.

Programmatic labels on LinuxArena. LinuxArena labels come from programmatic main-task tests and side-task detectors. The same pattern seen in vLLM holds here with review-dependent architectures reaching zero under collusion while the review-independent floor stays at 0.300.30 (Appendix H, Tables 11 and 12). Our mechanism trails pure routing on LinuxArena until routing collapses.

The vLLM results therefore measure how reviewers and the auditor respond to diffs written to implement a side task, and some of those diffs may not work when run. The theory, the audit floor and the collapse of review-dependent rules under collusion do not depend on this. Section 7 now discusses this limitation.

Weakness 2 and Question 4

W2: "Table 9's quoted feasibility margins use seven uniform audits and oracle true positives, whereas the headline mechanism reserves four audits and routes three."

Q4: "For the four-random/three-routed mechanism, what deterrence-participation interval remains when false negatives and false positives are considered together? ... a sensitivity analysis over the honest-value/deviation-gain ratio would clarify whether its feasibility conclusion transfers."

Response

We agree, and have extended Table 9 to the four-random/three-routed allocation with both error types. The interval remains open for both auditors, but our data certify it only for the independent auditor.

Table 1, at v=gv=gOwner-targeted intervalMarginCloses at v/gv/gCloses at ϕ\phi95% upper bound on ϕ\phiCollective
First auditor (3/102 false positives, 3/30 misses)[5.39g,17.2v][5.39g, 17.2v]3.2×3.2\times0.310.0700.074empty
Independent auditor (3/159, 8/113)[5.19g,29.0v][5.19g, 29.0v]5.6×5.6\times0.180.0720.0481.27×1.27\times, not certified

Reserved audits set the catch probability, q0=4(1−γ)/23q_0=4(1-\gamma)/23, while all seven audits set honest providers' false-positive exposure.

Sensitivity to v/gv/g. With negligible base payments the margin scales linearly with v/gv/g: for the independent auditor it is 2.8×2.8\times at v/g=0.5v/g=0.5 and 11.2×11.2\times at v/g=2v/g=2.

Caveat on the independent auditor's threshold. It is measured at 0.800.80, the 98th percentile of its suspicion on the first 102 benign diffs. We fixed that threshold before auditing a further 57 benign diffs, none of which it flags. At thresholds 0.340.34 and 0.670.67, its margin is 0.44×0.44\times and 1.48×1.48\times (Appendix L.2).

Thank you again for the review. Consolidating Table 9 at the headline allocation made the feasibility claim narrower and more accurate.

Thank you for the clarification. The updated calculation addresses my question about the headline configuration. Two empirical limitations remain: the vLLM evaluation measures responses to code labeled as malicious without confirming that the attacks work when executed; and the feasibility result holds at the selected auditor threshold, while the reported alternatives yield substantially different margins, including one with no feasible interval. These limit what the experiments establish about deployment, rather than invalidate the theoretical results. My overall assessment remains unchanged.

Reviewer_jTRoOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies how to verify artifacts generated by multi-agent systems when peer review may be untrustworthy (e.g., due to colluding reviewers) and trusted verification is limited. Given a limited trusted audit budget, the proposed strategy reserves part of the budget for an independent uniformly random draw over modules, while the remaining budget is routed via a diversity router towards unusually suspicious modules. The authors also assign a penalty relative to the random audit floor to dissuade unsafe modules by making them unprofitable for rational providers. On a testbed of 30 vLLM backdoors with 5 reviewers, majority voting misses 21 of them under honest reviews. Furthermore, as the size of the colluding coalition increases, review-dependent strategies eventually drop to no backdoors being caught. However, the proposed strategy maintains the guaranteed catch rate floor from the reserved random draw, and with honest reviews, their diversity routing raises the catch rate to 90.9%.

Critical strengths
  • The paper tackles an interesting practical problem in scalable oversight for multi-provider settings where no reviewer is trusted and the trusted audit budget is limited. The proposed mechanism is clearly presented and straightforward to implement, and empirical results demonstrate how their proposed diversity routing strategy can extract meaningful relative signal that elevates catch rate over uncalibrated raw reviews. Results also make the tradeoff between random floor and routing strategy clear as colluding coalition size varies, which is useful for practitioners to consider in real deployments.
  • The analysis of the problem setting is extensive, such as including further feasibility discussion relating acceptable penalties relative to honest deployment value and the trusted auditor’s false-positive rate, as well as attack surfaces such as a stress-grid of adversarial report strategies colluding reviewers could make with their reviews and mechanism-aware author. The strengths and limitations of the strategy in practice are made clearer as a result.
Critical weaknesses
  • It would be ideal to study the routing strategy more in-depth since that is the only step that can raise the catch rate over the random floor guarantee. Proposition 1 considers the any-flag routing strategy, but then immediately jumps to a diversity router strategy. How does a strategy such as just tracking how many flags a module receives then rank to achieve the top-k selection (ties broken randomly for example) do? What if the rubric from reviewers changes from assigning scores to simply applying binary flags? I’d also be particularly interested in how effective these baselines are depending on how the quality of the reviewing models varies (e.g., comparing the open models currently being used to a full suite of frontier model reviewers). I see an example of this already happening in the Appendix section on LinuxArena, so I think reviewer capability should be an axis generally explored in the setting presented in the main paper. If we have a certain level of trust of the capabilities of our reviewers, are different routing strategies empirically optimal?

  • The authors should discuss how a practitioner should set their budget for random floor vs. routing prior to kicking off an artifact project. For example, the budget (4 random, 3 routed) doesn’t actually seem to be optimal in some of the ablation experiments (LinuxArena, Table 12). Is there some approximate simulation one could run to tune this hyperparameter? Also, it would be nice to see how trends change as the number of modules increases if the authors have reasonable compute to do so.

Questions for authors
  • Can the authors include some practical discussion in the main paper on how g and v might be calibrated with respect to each other? For example, in the empirical setting described in the paper, some of the safety issues/backdoors being looked for (reviewer prompts in D.1) are high value targets like exfiltrating secrets or adding a hidden trigger, which could easily outprice the feasibility band ~7.5v if deployed (B.5). A penalty could exist relative to this very high backdoor gain, but then honest providers may not want to participate due to high costs in the case of a false positive. In general, it may be helpful to position this appendix participation analysis somewhere in the main work next to the deterrence claims in the main work for practical deployment considerations.

  • How would the analysis change with heterogeneous module gains (e.g., a module related to credential handling is probably more meaningful to an adversary than some logging tooling)? I see B.4 clarifies the scope for non-uniform random auditing, but I’m curious how the optimal audit floors strategy changes when the uniformity condition for minimax optimality doesn’t hold.

  • Is there a reason the method in 5.2 is random selection first, then router over the remaining budget? For example, what’s the tradeoff with the reverse method, route first then random select the remaining budget.

  • While the authors make the single-shot setting clear, can they also present some discussion about which results hold under repetition/numerous turns and which no longer apply? This would be especially helpful since multi-agent collaborations such as in the software setting typically go beyond single-shot in scale.

3 comments
AuthorsOpenReview ↗

Response to Reviewer jTRo (1/2): Weaknesses

Thank you for the detailed review. Your two weaknesses each led to new experiments, which we describe first. Changes are in green in the revised PDF.

Weakness 1

"How does a strategy such as just tracking how many flags a module receives then rank to achieve the top-k selection (ties broken randomly for example) do? What if the rubric from reviewers changes from assigning scores to simply applying binary flags? ... I think reviewer capability should be an axis generally explored in the setting presented in the main paper."

Response

We ran all three on an enlarged pool of 113 backdoors (Appendix K.2 and K.3).

Flag counting. We rank modules by the number of reviewers at or above a threshold and audit the top seven, with ties broken at random. Under honest review, a count at threshold 0.340.34 matches the diversity router, and a count at 0.670.67 loses over a quarter of the catch. Against one colluder, the router leads the better count by 0.090.09 [0.04,0.14][0.04, 0.14]. Counting is weak once colluders also flag their own safe modules (decoys):

Table 1Honest reviewFour colluders, with decoys
Flag count (≥0.34\geq 0.34)0.930.02
Diversity router0.940.13
Ours (4 random + 3 routed)0.930.27

Binary flags. Feeding the router each reviewer's binary flag in place of its score reproduces the count at 0.340.34 in every cell under suppression. We computed the flags by thresholding the cached scores and did not re-elicit them with a binary rubric. We show this in Appendix K.2.

Reviewer capability. With five provider-distinct frontier reviewers, majority voting under honest review rises from 0.240.24 to 0.830.83. Pure routing then survives a colluding majority (0.850.85 with three colluders), falls to 0.470.47 with four, and reaches zero with five. With this panel our reserve costs 0.130.13 [0.07,0.18][0.07, 0.18] of catch probability at three colluders and 0.270.27 [0.19,0.36][0.19, 0.36] at four, so our mechanism leads pure routing only under full collusion (4/234/23 against 00).

Overall, the answer to your last question is yes. The best allocation depends on reviewer capability. A strong panel favours a smaller reserve unless full collusion is the primary threat. Section 6 now reports this.

Weakness 2

"The authors should discuss how a practitioner should set their budget for random floor vs. routing prior to kicking off an artifact project... Is there some approximate simulation one could run to tune this hyperparameter? Also, it would be nice to see how trends change as the number of modules increases."

Response

We added a three-step procedure for practitioners and ran a sweep to demonstrate tuning (Appendix K.5). The procedure:

  1. Minimum reserve. The honest-participation ceiling sets a lower bound. At v=gv=g, it is two of seven slots with the first auditor and one with the independent auditor.
  2. Maximum reserve. The largest reserve that loses at most 0.030.03 of catch under honest review. This gives five slots on both pools. Note: the 0.030.03 tolerance is a design choice that we made after seeing the sweep.
  3. Choose within that range based on the reviewers you have. With our open-weight reviewers, each extra reserved slot (up to five) catches more backdoors when most reviewers collude. With stronger frontier reviewers, fewer reserved slots work better.
Table 2, reserved slots of 702456
Honest review0.940.930.930.910.81
Three colluders0.370.420.470.500.38
Audit floor0.000.090.170.220.26

Our choice of four slots lies inside the range and is not uniquely optimal. We agree with your LinuxArena observation. There, four reserved slots lower honest catch from 0.820.82 to 0.770.77, which exceeds the tolerance, so the procedure caps the reserve below four.

The sweep provides a simulation that a practitioner can use to tune the split. We run the sweep by collecting the reviewer scores once, then re-run the audit draw on them for each split. This way we can tune without needing to collect new reviews.

Module count (Appendix K.4). We compose artifacts of M=23M=23 to 103103 modules at a fixed audit fraction of 0.30.3. Honest routing stays between 0.920.92 and 0.950.95, and routing against a colluding majority stays at 0.370.37 for every MM. Our mechanism's honest catch drifts from 0.930.93 at M=23M=23 to 0.880.88 at M=83M=83. At a fixed number of reserved slots, the penalty grows linearly with MM because the floor is krand/Mk_{\mathrm{rand}}/M.

AuthorsOpenReview ↗

Response to Reviewer jTRo (2/2): Questions

Question 1

"Can the authors include some practical discussion in the main paper on how g and v might be calibrated with respect to each other? ... it may be helpful to position this appendix participation analysis somewhere in the main work next to the deterrence claims."

Response

We agree, and have added a calibration paragraph to Section 5.1 next to the deterrence claim, with the full analysis in Appendix B.5.

With negligible base payments, feasibility depends only on the ratio v/gv/g; our v=gv=g is a normalisation and not an empirical estimate. At the four-random/three-routed configuration with measured audit errors, a feasible penalty requires g≲3.2vg\lesssim3.2v with the first auditor and g≲5.6vg\lesssim5.6v with the independent auditor. Both are tighter than the 7.5v7.5v you cite, which uses the pure seven-slot floor without audit misses.

In practice, the principal chooses an upper limit on gg (the most a provider could gain from one unsafe module) and sets the penalty for that limit. Restricting module privileges lowers the limit. For targets whose gain exceeds the threshold, as secret exfiltration may, no penalty both deters and preserves honest participation. The reserved draw still inspects each module with probability 4/234/23 and blocks deployment when the auditor confirms the violation.

Question 2

"How would the analysis change with heterogeneous module gains? ... I'm curious how the optimal audit floors strategy changes when the uniformity condition for minimax optimality doesn't hold."

Response

The penalty-minimising floor weights modules by their gain bounds (new Proposition 7, Appendix I). With gain bound gjg_j and auditor miss probability γ\gamma, the minimum penalty Φ∗\Phi^\ast solves ∑jgj/((1−γ)(gj+Φ∗))=krand\sum_j g_j/((1-\gamma)(g_j+\Phi^\ast))=k_{\mathrm{rand}}.

For example, with two credential-related modules bounded at 10g10g and 21 modules at gg (krand=4k_{\mathrm{rand}}=4 with a perfect auditor), uniform auditing requires 47.5g47.5g and gain-weighted auditing 6.53g6.53g.

The gain bounds must be fixed independently of reviewer reports, since reviewer-assigned importance would let colluders steer the guaranteed allocation. Without reliable module-specific bounds, the uniform floor remains the minimax choice. Appendix B.4 now states this.

Question 3

"Is there a reason the method in 5.2 is random selection first, then router over the remaining budget? For example, what's the tradeoff with the reverse method?"

Response

Neither order dominates, so we now report both in the revision (Sections 5.1 and 5.2, Appendix J, Table 18).

We chose random-first because the reserved draw is then independent of reports, which gives the bound q0=krand/Mq_0=k_{\mathrm{rand}}/M directly. Route-first has the stronger worst-case guarantee at the same budget as the four random audits are drawn from the 20 unrouted modules, so the floor is 4/204/20 against 4/234/23, and with a perfect auditor the minimum penalty falls from 4.75g4.75g to 4g4g.

Route-first loses near the routing cutoff. Under random-first, a random audit of a highly ranked module frees a routed audit for the next-ranked module, so random-first catches about 0.0450.045 more under partial suppression by two or three colluders. All other experiments use random-first.

Question 4

"While the authors make the single-shot setting clear, can they also present some discussion about which results hold under repetition/numerous turns and which no longer apply?"

Response

We added this to Section 7 and Appendix I.

  • Holds over rounds. The audit probability and the threshold of Proposition 3, because the mechanism draws a fresh random audit for each artifact, independently of reports and history.
  • Weakens over rounds. Feedback across rounds can reveal which attacks evade routing, which drives catch probability toward the floor, and which attacks the auditor misses, which raises the effective miss rate γ\gamma.
  • No longer applies over rounds. Our model has no reputation term, so it does not establish equilibria of the repeated game, and it gives no extra leverage against a provider that behaves for many rounds and plants once. Violations staged across artifacts are not covered.

Thank you again for your feedback. The routing baselines and reserve-setting procedure are a positive addition to the paper, and we appreciate the recommendation.

I appreciate the authors' thorough response and experiments. My weaknesses and questions are largely addressed.

I’m not sure I completely follow the proposed hyperparameter tuning procedure. It seems that deriving the maximum reserve (steps ii–iii) requires a calibration round in which the same reviewer panel scores modules with known ground truth. In practice, would you have to calibrate on one project then extrapolate the same hyperparameters when you start a new project? How robust is this transfer?

That being said, I still raise my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.