- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces DF-Audit, a fairness auditing framework for LLM decision-making in hiring that generates pairwise comparisons from O*NET occupational requirements. The legitimate evidence fixes the correct ordering by construction, independently of any protected information. The framework separates evidence-following (within-group order preservation, OPR_within) from dependence on protected information (cross-group contrast ∆c and direct/proxy-enabled counterfactual interventions τ), and adds configurable stress tests that vary the number of evidence items, the differing items, the signal strength, and the presence of conflicting evidence. Across seven LLMs and four binary attributes, the authors report three findings: dependence persists in models that otherwise follow the evidence well (mostly favoring younger and U.S. candidates), group-level and intervention-based signals rarely agree, and harder configurations reveal dependence that the main audit does not detect.
Critical strengths
The construction in §4.2 and Appendix B.1 yields an unambiguous ground-truth ordering, since the winner dominates the loser on every requirement and R is generated before X. The invariant is also verified after generation (Appendix B.5).
The diagnosis matrix (Table 3) reads |∆c| against the within-group error 1 − OPR_within, which gives a principled way to distinguish “protected information overrides usable evidence” (GPT-4o-mini, age) from “protected information breaks ties among unresolved pairs” (Qwen3-14B).
The statistical treatment is careful: paired sign-flip tests with Holm correction, a δ-sensitivity analysis (Table 12), a tie-permitted robustness check (Table 14), and suitably hedged discussions of what is not identified (Appendix D.4 on indirect effects, Appendix E.3 on proxy-only presentation).
The stress tests produce concrete, non-obvious results. GPT-4o-mini’s age gap becomes sustained only from 20 evidence items onward, while OPR_within remains at or above roughly 0.95 (Table 16, Fig. 3a,d). GPT-4.1, which is clean in the main audit, shows a significant preference for younger candidates at five of the six conflicting-evidence levels (Table 17).
Critical weaknesses
The “one signal is not enough” finding (Obs. 2) is likely in part an artifact of the intervention design. Interventions use identical-evidence pairs with ties permitted, and several models simply tie every such pair: Qwen3-14B has τ = 0 and 0% flips on all attributes (Tables 4 and 13), and GPT-4.1 ties all 200 units in Table 17(iii). In these cases τ is structurally floored rather than independently informative. Interventions on pairs with a small legitimate gap, or under forced ranking, would test whether the two views genuinely diverge.
The main configuration is highly artificial: 40 templated items, a single “slightly above” vs. “slightly below” difference, and an explicit attribute line (“Race: white”, “The candidate is 62 years old”) placed next to the proxy (Fig. 4). The dominant age finding may partly reflect models treating stated age as job-relevant under the instruction to “use job-relevant evidence,” rather than discrimination in realistic screening. Adding a free-text résumé condition, and extending the proxy-only analysis beyond race and two models (Appendix E.3), would strengthen external validity.
Each prompt is queried once and some decoding is non-deterministic (Appendix C), yet no noise floor is measured. Re-querying unchanged prompts would show how much of the flip rates and the small ∆c values (e.g. Grok-3-mini age, −1.4 pp) exceeds run-to-run variability. Decoding settings are also not reported for GPT-4.1 (main audit), MiniMax-M3, Gemini-3.5-Flash-Lite and Qwen3.7-Plus, which limits both interpretation and reproducibility.
The stress testing that motivates the “dynamic/scalable” framing covers only two models with hand-chosen sweeps, and the conflicting-evidence target depends on an importance-weighted score that the prompt never specifies, with acceptance margins as small as 0.002 (Appendix B.1). Where OPR_within is 0.55–0.75, the “legitimate” ordering is partly an author convention. Running the sweeps on all seven models and reporting results as a function of the margin would make Obs. 4 more convincing.
Questions for authors
See weaknesses