acceptodds
Desk rejected at ICLR 2027

ASSUMPTION-STRATIFIED AUDITING OF DETERMIN-ISTIC EVALUATORS

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Correct observed labels do not identify an evaluator’s behavior on valid counter-factuals, and a ffnite class of plausible faults is not exhaustive merely because it ffts those labels. We formulate small-budget auditing of deterministic evalu-ators as uncertainty over a fault’s witness set: the probes on which it disagrees with an independent semantic oracle. We separate three assumptions: a failure resembles a modeled survivor; some frontier probe reveals it; or multiple wit-nesses occur inside a visible contract mechanism. Our central result identiffes what optimal unstructured protection leaves unresolved. At budget two, it ffxes how often each probe is queried but not which probes are queried together, yield-ing an exact range of detection probabilities for the same multi-witness fault. The mechanism-supported assumption constrains part of that remaining freedom, and our allocator balancesthe registered assumptions. Acrosssix frozen IFBench eval-uators, the resulting cases include a free row-41 reffnement, costly reffnements for no whitespace and sentence:keyword, and frontier insufffciency under the modeled assumption for three tasks. Frozen hidden-fault validation is mixed

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
auditor identifiability, witness-set assumptions, deterministic evaluator auditing, audit policy dimension, assumption-strength phase transition, partial-identification rank gain

Related papers

Discussion (0)

Sign in to comment.