A Resource-Based Framework for AI Auditing: Access, Data, and Compute

AI audits have garnered signficant attention in recent months, as the ability to test and monitor AI models plays a crucial factor in safe development and deployment. However, AI audit success hinges on an auditor's capabilities; in particular, what information and resources they have at their disposal. Ongoing debates have surrounded the question: *How much model access is needed to conduct an AI audit?* Auditors seek more model access, while companies resist it for reasons related to confidential business information (trade secrets) and privacy. This tug-of-war remains unresolved. Further, this debate misses a related question: *How much data and compute do auditors need?* In this work, we systematically investigate both questions. We provide a framework that varies each of these three dimensions—model access, data, and compute—and measures their joint effect on audit success. We apply this framework to three tasks. We find that (1) some model access is important, but there are strong diminishing returns; (2) data and compute matter more for traditional fairness audits, but model access matters more for manipulation and lie detection; and (3) these three resources can be fungible, meaning that model access is not the only salient factor. These results suggest that audit governance should jointly characterize (access, data, compute) requirements in relation to audit objectives. Further, our framework serves as a template that can be applied to different audit tasks and inform the degree of auditor authority.