DF-Audit: Scalable Fairness Auditing for Disentangling Legitimate-Evidence Use from Discriminatory Reliance in LLMs

LLMs are increasingly used in socially consequential decision-making settings such as hiring, raising concerns about differential treatment across groups defined by protected attributes (e.g., race, nationality, and gender). Existing fairness benchmarks typically generate a fixed set of instances that compare model decisions under interventions on such protected information while holding other information fixed. However, observed decision outcomes alone cannot determine whether a decision error reflects inadequate use of legitimate evidence or sensitivity to protected information, while predefined test conditions provide limited control over decision difficulty, making it difficult to stress-test increasingly capable models. We introduce DF-Audit, a dynamic fairness auditing framework that disentangles failures in legitimate decision making from dependence on protected information through controlled contrasts. We formulate decision quality as a pairwise ranking problem and progressively introduce protected-information variation for increasingly fine-grained fairness diagnosis. Within-group comparisons first establish whether the model preserves rankings implied by legitimate evidence; cross-group comparisons then test whether the same legitimate ordering changes with protected-group composition; and counterfactual interventions finally hold legitimate evidence fixed while manipulating protected information to probe finer direct and proxy-enabled pathways. Beyond diagnosis, DF-Audit scales the audit through a breadth-to-depth process: breadth explores diverse configurations and randomly sampled instances across legitimate and protected-information conditions, while depth progressively increases difficulty within selected conditions by lengthening the context, narrowing differences in legitimate evidence, increasing conflicts among legitimate cues, and strengthening protected-information challenges.