DynaAudit: Auditing Hidden Supervision-Channel Drift

Weak-supervision systems are often monitored through features and source outputs, yet the mapping from those observables to the latent label can change after deployment. We study hidden supervision-channel drift, where distinct supervision mechanisms induce the same declared unlabeled monitor transcript while implying different gold-label posteriors. This creates a fundamental blind spot: under exact transcript equivalence, purely unlabeled detection has power no greater than its false-alarm rate, and unlabeled estimation incurs irreducible error. We characterize support auditability—whether trusted labels available on the deployment support can distinguish candidate mechanisms—and, for a finite correctly specified candidate family, derive an anytime finite-sample certificate for fully adaptive auditing based on accumulated pairwise Bhattacharyya information. The guarantee is acquisition-policy agnostic, separating statistical validity from acquisition speed. These results motivate DynaAudit, which separates representative sentinel detection, mechanism-aware diagnosis, model criticism and abstention, and inclusion-aware reuse of adaptively selected labels. Controlled and temporal experiments show that targeted audits concentrate diagnostic information when post-alarm labels are scarce, while matched-budget random auditing can catch up as total trusted-label budgets grow. Natural temporal data further show that prediction can improve even when no named mechanism is defensibly supported. Overall, DynaAudit characterizes when unlabeled monitoring is fundamentally insufficient and what additional trusted evidence is required for auditable adaptation.