First-Order In-Context Shapley: Scalable Data Auditing for LLM Alignment and Evaluation

Post-training alignment of Large Language Models (LLMs) is increasingly bottlenecked by data quality: massive preference and instruction-tuning corpora inevitably accumulate label noise, safety inversions, and inter-annotator contradictions that surface-level deduplication cannot detect and that are too costly to find by LLM-judging every record. While Data Shapley provides a principled framework for valuing individual records, exact computation requires intractable retraining, and fast embedding-space surrogates such as KNN-Shapley fail to separate high- from low-utility demonstrations ($\rho =0.0003$ with generative utility). We introduce an inference-only valuation pipeline that computes the first-order term of an in-context cooperative game over semantic $k$-NN neighborhoods in $O(k)$ forward passes without parameter updates, achieving positive rank correlation with multi-shot Monte-Carlo in-context Shapley on $100\%$ of targets (mean $\rho =0.66$, close to the estimator's own split-half agreement of $0.73$) with $19\times$ less compute. By projecting pairwise log-likelihood shifts onto a directed influence graph, we characterize each record through three signals: outgoing Exemplifying Advantage, incoming Exemplified Advantage, and a topological One-of-a-Kind Score that protects rare tail records. Pre-filtering with these signals restricts arbitration to $1\%\text{–}5\%$ of records before routing each candidate and a contrasting twin to an LLM arbitrator, achieving $4.5\times$ enrichment in arbitrator-confirmed annotation errors on graded response-generation data (HelpSteer2) and $2.2\times$ on pairwise preference data (Anthropic HH-RLHF), including contradictions that zero-shot likelihood and margin filters miss. During DPO training, confirmed preference contradictions oppose the realized update far more often than clean records from early on ($56.7\%$ vs. $37.0\%$ in exact LoRA gradients) and stay below chance in reward accuracy. Finally, auditing the HH-RLHF evaluation benchmark exposes label errors on which aligned models disagree with the benchmark $73\%\text{–}83\%$ of the time ($87\%\text{–}95\%$ when three frontier arbitrator families unanimously agree), and retraining DPO from scratch after repairing or filtering the flagged training records improves reward accuracy by $+0.5\%\text{–}0.6\%$ overall and $+3.1\%\text{–}3.5\%$ on the repaired test inversions, even without the LLM arbitration step.