$\Delta$-Audits: Adaptive Performance Monitoring

Evaluations are the cornerstone of ML governance. Audits of ML-enabled systems are conventionally treated by regulators and academics as one-shot events: a single audit is conducted, its outcome is certified, and the system is presumed compliant until a fresh audit is commissioned. However, deployed systems are updated regularly, often without the auditor's knowledge, which can invalidate the audit's conclusions. To help regulators choose when to commission a fresh audit, we introduce \deltaaudits, a setting that pairs the initial white-box audit with a subsequent lower-access black-box monitoring phase. We instantiate monitoring with a fixed-budget disagreement test parameterized by a query distribution. We use the initial white-box inspection phase to optimize this distribution through a bi-level formulation that anticipates a provider maximizing its utility under the monitoring constraint. The goal is to limit deterioration in auditor utility while leaving the provider room to improve its own utility. Experiments on tabular and image classification tasks evaluate manipulation gain under uniform and optimized monitoring constraints. Optimized monitoring detects harmful updates with an order of magnitude fewer queries than uniform or performance-based monitoring, and sharply reduces the utility a provider can gain by manipulating the audit, whereas uniform monitoring leaves this gain almost intact.