acceptodds
Under review as a conference paper at ICLR 2027

Earned Redundancy: Auditing LLM Skill-Promotion Quorums

Abstract

Promotion gates combine language-model reports to decide whether reusable agent skills can be admitted. Different model names, prompts, or endpoints do not establish that agreement reduces malicious admission while retaining benign skills. We introduce an intervention-conditioned audit that separates observer identity from acquired evidence, measures joint failure, and tests decision benefit against a declared reference under an explicit missing-report policy. On an identity-disjoint panel of 120 malicious packages, the sole preregistered dependence contrast yields (95% interval , one-sided ). Yet the least dependent pair jointly clears 14/120 packages, versus 13/120 for the more dependent prompt pair, because their error margins differ. In calibrated simulation, target clones reverse the sign of proxy-adjusted dependence. A paired evidence-view audit changes joint clearance from 16/28 to 0/28 on an outcome-selected cohort. A benign-inclusive Claude–Qwen extension finds lower raw cross-family correlation but no resolved balanced-accuracy gain for the calibration-selected rule, with the dependence advantage sensitive to marginal-rate normalization. Redundancy credit therefore requires a declared intervention, evidence view, and measured risk-utility improvement against the reference.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM safety, skill promotion, ensemble reliability, correlated failures, evidence acquisition, risk–utility tradeoffs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BcjsOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is clearly AI written and not very understandable. My best understanding is that the authors [or their AI] are investigating how LLM skills can be audited for malicious content and to what extent ensembling helps and how it depends on ensemble members being from same / different families. As one would expect, different families help all else equal, but because in real world all else is generally not equal, they find that it does not actually always help. The paper is trivial and lacks any non-trivial scientific insights imo and should be rejected.

Critical strengths

No strengths. This is an AI slop paper that should be strongly rejected.

Critical weaknesses

The paper is AI written and incomprehensible to me as a human reader

The paper is not readable and seems like AI generated. I am highlighting specific issues just within first section of section 2.2 PROBLEM FORMULATION (pasted below):

A promotion service admits reusable skills to an agent library using observer reports about fixed artifacts. Later execution is evaluated separately. Let C be a candidate skill and Y ∈ {0, 1} its evaluation label, with Y = 1 denoting malicious content that should not be promoted. Observer i receives a registered view Xi(C) and reports clear, protected, or insufficient. Let Vi = 1 denote passing schema, cardinality, identifier, and binding checks. Complete-clear requires Vi = 1, complete coverage, untruncated input, and a clear verdict. The formal fail-closed map sets Ai(C) = 1 for complete-clear and 0 otherwise.

  1. What is a promotion service?
  2. What is an observer report?
  3. "Later execution is evaluated separately" -> air-dropped sentence with no connection to surrounding context and unclear meaning. No explanation of why is this evaluated separately..
  4. Bunch of symbols are dropped without any clear explanation of their context and how they are related.
  5. lots of nouns with ambiguous meaning; cardinality, what is it referring to here? what is an identifier? what is binding check?

This is not at all comprehensible to me (a human). I have read the problem statement multiple times and can not say I understand it in a satisfactory way. And to stress this issue is widespread through the whole paper.

There is no clear research contribution made by the paper

The paper has no clear contribution / findings in my view. The paper is, as most AI slop papers do, projecting some run-of-the-mill results as non trivial (e.g., propositions 1 and 2 in the paper as far as I can tell follow straightforwardly from the modelling assumptions, which seem to be non-trivial and insufficiently justified, and also these propositions do not contribute anything).

The figures are AI generated and AI usage statement is incorrectly claiming otherwise

AI usage statement says AI was not used in preparation of figures, but that is clearly false. My sense is that even experimental design etc. was also done by AI and AI usage statement is clearly misrepresenting / underreporting AI use.

Questions for authors

N/A

Reviewer_iBkZOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether using multiple LLM observers improves the safety of admitting reusable agent skills. The authors propose an audit protocol that distinguishes observer identity, the evidence available to each observer, and the dependence between their errors. They argue that lower error correlation does not necessarily reduce the probability of jointly accepting malicious skills, and that any benefit from redundancy should be evaluated against a reference observer while accounting for benign skills that are incorrectly rejected.

The experiments use MaliciousSkillBench and several LLM observers, with a preregistered dependence comparison on 120 malicious packages. The authors find that cross-family observers have lower error correlation than same-model prompt or temperature variants, but do not necessarily achieve lower joint false-clear rates. A separate Claude–Qwen evaluation finds no established balanced-accuracy gain from combining the two models. The paper also examines bias from using related observers to estimate item difficulty and shows that expanding the package text available to fixed observers reduces joint clearance on a selected set of malicious packages.

Critical strengths
  • Careful experimental design and statistical analysis. The paper goes beyond reporting pairwise agreement. It measures marginal false-clear rates, joint errors, benign acceptance, and uncertainty under explicit missing-report policies. The preregistered identity-disjoint experiment is a strength, and the authors are generally careful not to present descriptive comparisons as confirmatory findings. The comparison between the prompt pair (13/120 joint false clears) and the cross-family pair (14/120) provides a concrete example of why error correlation alone is not sufficient to assess an admission rule.

  • Useful analysis of failure dependence and evidence availability. The difficulty-proxy experiments show how using related observers in an adjustment procedure can distort residual correlation estimates. I also find the paired package-view experiment interesting. Holding the observers and manifests fixed while changing the available text provides evidence that inspection coverage matters. These results are relevant to the design of agent skill auditing systems, where different observers may share both failure modes and limitations in the evidence they receive.

Critical weaknesses
  • The methodological novelty is somewhat limited, and the empirical results do not yet establish a clear practical advantage of the proposed audit framework. The relationship between marginal error rates, error correlation, and joint failure probability is already well understood in ensemble reliability. Similarly, the balanced-accuracy comparison against a reference follows directly from the definitions of malicious blocking and benign acceptance. The paper applies these principles carefully to skill promotion, but it is less clear what new methodological capability the three-layer audit provides beyond evaluating observer combinations on labeled data. The empirical results mostly identify cases where redundancy does not help. For example, in the balanced Claude–Qwen experiment, the selected pair clears the same 11 malicious packages as the reference while rejecting two additional benign packages. This is a useful negative result, but the paper does not demonstrate that the audit can prospectively identify a better promotion configuration, or establish when adding a second observer is preferable to improving evidence coverage for a single observer. I think a direct comparison of alternative auditing configurations under the same evaluation protocol would make the contribution stronger.

  • The experimental evidence is not sufficient to establish how well the proposed audit transfers to practical promotion decisions. The main preregistered experiment tests one dependence contrast involving specific prompt, temperature, and cross-family configurations. It supports the reported ordering for these observers, but provides limited evidence about model diversity more generally. This matters because the balanced Claude–Qwen extension shows that the apparent cross-family dependence advantage becomes unresolved after normalization for attainable correlation bounds. The decision-level evaluation is also less conclusive than the registered dependence test: the calibration-selected operating points were reconstructed after the formal outputs existed, and the authors correctly treat this as a supplementary analysis. The package-view experiment has a different limitation. It studies 28 previously misclassified malicious packages, without a corresponding benign evaluation. The reduction in joint clearance from 16/28 to 0/28 is interesting, but does not show whether expanded inspection improves the overall promotion decision or how it compares with adding independent observers. As a result, the three parts of the audit are individually informative, but their combined value for selecting a deployment configuration remains unclear.

Questions for authors
  • Can the authors evaluate observer diversity and evidence expansion under the same decision-level protocol? For example, I would be interested in comparing a single observer with full-package access, two observers with primary-file access, and two observers with full-package access on the same malicious and benign evaluation set. This would help determine whether an additional observer provides value beyond improving what the existing observer can inspect.

  • How robust is the dependence ordering when observer error rates are comparable? The confirmatory experiment finds lower raw correlation for the cross-family pair, while the Claude–Qwen extension shows that the average advantage becomes unresolved after marginal-rate normalization. Can the authors provide a comparison at matched operating points, or explain what conclusions about model diversity remain supported across the two experiments?

2 comments

Dear Reviewer,

As the deadline for the discussion period approaches, we kindly request your feedback on our response. Your insights would be greatly appreciated.

Thank you for your time and consideration!

Thank you again for your thoughtful review and for the time you have devoted to our paper. Your suggestions have helped us improve it considerably.

We have responded to each of your concerns in our rebuttal, and the revised manuscript marks all added content in blue. As the discussion period is coming to a close, we would be sincerely grateful if you could take a brief look at our responses. If they have addressed your concerns, we would greatly appreciate it if you would consider reflecting this in your final evaluation. Any further feedback would also be very welcome.

Thank you very much for your time and consideration.

Reviewer_iaxaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies whether combining LLM inspection reports earns additional assurance for admitting reusable agent skills. It separates observer identity, supplied evidence, and joint failure, and evaluates admission rules against a reference using malicious false-clear risk and benign acceptance. A preregistered, identity-disjoint malicious panel confirms a specified dependence contrast, while descriptive joint-clear counts illustrate why lower correlation need not imply lower joint risk when marginal error rates change. Simulation demonstrates distortion from target-contaminated difficulty proxies. A paired expanded-text intervention changes verdicts on a selected historical failure cohort, and a benign-inclusive extension finds no statistically resolved balanced-accuracy improvement for its calibration-selected pair.

Critical strengths

Useful decision-level accounting rather than a diversity proxy. The paper distinguishes error association from joint admission risk and from reference-relative decision benefit. Equations 7–9 make these distinctions explicit, and the balanced extension provides a concrete example: the selected pair recovers none of the reference's observed malicious errors while blocking two additional benign items. Reporting both safety gains and benign losses is more informative than interpreting lower correlation as assurance. [§3, p. 5; §5.3, p. 8; Tables 10–11, p. 22]

Careful disclosure of evidential scope and uncertainty. The manuscript distinguishes preregistered confirmation, selected-cohort intervention, and post-observation diagnostics; acknowledges that the empirical residual sign change also changes the item panel; and supplements a degenerate zero-recovery bootstrap with a conservative component-based interval. The report-handling sensitivities and explicit selection chronology make the results unusually inspectable. [§5.2–5.4, pp. 7–8; §6, p. 9; Appendix B.5–B.7, pp. 17–19]

Critical weaknesses

Limited methodological novelty and demonstrated decision benefit temper the significance of the contribution. The covariance and balanced-accuracy decompositions are elementary identities, and the paper appropriately situates strength/dependence accounting in prior multiversion and ensemble research. The balanced extension supports a useful accounting demonstration, but does not establish a prospective deployment benefit. Its reconstructed operating-point chronology is disclosed and does not, by itself, invalidate the disjoint-calibration comparison. The decisive reservation is how much new capability the intervention-conditioned audit adds beyond established ensemble evaluation, rather than a missing control required for the stated diagnostic claims. Clarifying that increment would strengthen the contribution; prospective validation would extend its scope. [§2.1, p. 3; §3, p. 5; Appendix A.1, p. 15; §6, p. 9; Appendix B.5, pp. 17–18]

The view intervention has limited explanatory and practical reach, as the authors acknowledge. It demonstrates verdict changes in 28 selected malicious packages, not whether expanded text improves semantic detection rather than increasing conservatism. The manuscript expressly disclaims benign utility and independent malicious-path adjudication. This is not an unaddressed flaw in that bounded demonstration, but limits the strength of the mechanistic and deployment insight supporting acceptance. Benign-inclusive adjudication or targeted ablation would extend the contribution rather than serve as a necessary repair of the current diagnostic claim. [§5.4, p. 8; Appendix B.6, p. 18; Table 14, p. 23]

Questions for authors

What specific methodological or operational capability of the intervention-conditioned audit is not already supplied by established ensemble strength/dependence evaluation and paired risk–utility comparison? Please distinguish the new audit contract from the elementary accounting identities. [§2.1–3, pp. 3–5] Can the original calibration-selection rule and its chronology be independently checked in the supplementary record, notwithstanding the absence of operating-point names from the historical pre-formal seal? What exactly was fixed before formal outputs existed? [Appendix B.5, pp. 17–18]

1 comment

Dear Reviewer,

As the deadline for the discussion period approaches, we kindly request your feedback on our response. Your insights would be greatly appreciated.

Thank you for your time and consideration!

Comments on the paper

2 comments
AuthorsOpenReview ↗

Global Response(1/2)

Dear reviewers, thank you for your thoughtful and constructive feedback!

Our Strengths Summarized by Reviewers

We sincerely appreciate the reviewers for highlighting the following strengths of our work:

  • Decision-level accounting rather than a diversity proxy. The audit separates error association, joint admission risk and reference-relative decision benefit, and explicit equations make these distinctions precise. The balanced extension gives a concrete example. Reporting both safety gains and benign losses is more informative than reading lower correlation as assurance (Reviewer iaxa, Reviewer iBkZ).
  • Careful experimental design and statistical analysis. The preregistered identity-disjoint experiment measures false-clear rates, joint errors and benign acceptance under explicit missing-report policies. The 13/120 versus 14/120 comparison shows why error correlation alone cannot assess an admission rule (Reviewer iBkZ, Reviewer iaxa).
  • Careful disclosure of scope and uncertainty. The paper separates preregistered confirmation, intervention and diagnostics, and does not present descriptive comparisons as confirmatory. A conservative component-based interval, report-handling sensitivities and an explicit selection chronology make the results unusually inspectable (Reviewer iaxa, Reviewer iBkZ).
  • Useful analysis of failure dependence and evidence availability. The difficulty-proxy experiments show how related observers in an adjustment procedure can distort residual correlation estimates. The paired package-view experiment holds observers and manifests fixed while changing the available text, showing that inspection coverage matters for agent skill auditing systems (Reviewer iBkZ).
  • Sound and informative audit. The paper raises no substantive soundness concerns (Reviewer iaxa). Its audit layers are individually informative and give a substantially deeper understanding of the problem (Reviewer iBkZ).
AuthorsOpenReview ↗

Global Response (2/2)

Our Main Revisions and Supplementary Experiments

In response to the reviewers' feedback, we conduct the following new experiments and revise the manuscript accordingly:

  • Same-Panel Comparison of Evidence and Observers (Reviewer iBkZ, Reviewer iaxa)
    • We add a study on the same 300 malicious and 300 benign packages that compares one observer with full-package access against two observers with primary-file or full-package access (Section 4, Section 5.3, Table 1, Appendix C.6). One Claude observer with the full package reaches 93.83% BA, 9.17 points above the primary-file Claude-Qwen pair (Holm p = 0.000137). Adding Qwen to it blocks no further malicious package and rejects 37 more benign ones. Under equal per-package budgets, the single observer still leads with 91.67% versus 85.00% BA.
  • Robustness of the Model-Diversity Conclusion (Reviewer iBkZ)
    • We add a third model family, GPT-6.1 (Section 5.3, Appendix C.6, Table 16). Qwen and GPT-6.1 each share both malicious admissions of the full-package Claude observer and 26 of primary Claude's 29. Adding GPT-6.1 to that observer rejects 39 more benign packages (Holm p = 0.00301). A second family removes 3 malicious admissions with SKILL.md and none with full packages, while full packages remove 24 of the pair's 26. The audit therefore measures diversity by the reference errors it recovers, not by family identity or correlation rank (Section 6).
  • Prospective Selection and When a Second Observer Helps (Reviewer iBkZ, Reviewer iaxa)
    • Before any formal request, the sealed calibration scores ranked each pair below the single Claude observer in both views. The 600 formal packages confirmed both rankings (Section 6, Appendix B.5).
    • Section 3 extends Proposition 2 to declared costs. If one malicious admission costs as much as ww benign rejections, the pair is preferable exactly when wGM>LBwG_M > L_B. With primary files the pair helps only if w>8.7w > 8.7. With full packages no cost ratio favors it. Section 6 turns these results into three rules for a skill registry.
  • What the Audit Adds beyond Ensemble Evaluation (Reviewer iaxa, Reviewer iBkZ, Reviewer Bcjs)
    • Ensemble statistics take the observers' votes as given and cannot separate approvals given with and without a package's script (Sections 1 and 2.1). The audit contract fixes and checks four inputs: the evidence each observer received, the report map that turns each report into a vote, a reference declared before formal outputs, and a difficulty proxy built only from observers outside the tested families. In Section 6, evidence, dependence and decision form one selection procedure.
  • Selection Chronology (Reviewer iaxa, Reviewer iBkZ)
    • Appendix B.5 now lists what was fixed before any provider response: the calibration/formal split, the selection rule, the report map and the analysis. It quotes the rule, and replaying it on the retained calibration reports reproduces Claude A and Claude A-Qwen3.8 A.
  • More Evidence Changes Verdicts in Both Directions (Reviewer iaxa, Reviewer iBkZ)
    • A more conservative observer could only reject more. With the full package, Claude newly blocks 28 malicious packages, each citing a file other than SKILL.md, and newly accepts 14 benign ones (Section 5.4). Its malicious admissions fall from 29 to 2 of 300, while its benign acceptance rises from 260 to 265.
  • Clearer Problem Formulation (Reviewer Bcjs)
    • Section 2.2 is rewritten in plain language. It defines skill, promotion gate, observer and observer report before use, introduces the symbols in workflow order, and illustrates the four report checks with a real package. Section 3 explains why admission and execution are evaluated separately. Terms are unified across the paper.
  • Role and Assumptions of the Propositions (Reviewer Bcjs)
    • Section 3 states that both propositions are exact decompositions. They need only a fixed report map and error rates strictly between 0 and 1, without independence or equal-strength assumptions. Section 5.3 applies them to every pair in the new study.
  • AI Use Statement and Figure Captions (Reviewer Bcjs)
    • The AI use statement now covers all uses: language polishing, literature search, visual polishing of figures the authors drew, and help in implementing and analyzing the experiments added during the discussion phase. The captions of Figures 2 and 3 are simplified.

Together, these revisions answer every question and resolve every concern raised by the reviewers. The same-panel study shows that the audit selects the better promotion configuration before deployment. The revised text makes the formulation, the assumptions and the AI use clear.

We thank all reviewers again for helping us strengthen the decision-level evidence, the clarity of the formulation and the presentation of the paper.

A revised draft of our manuscript has been uploaded, with added content marked in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.