Beyond Local Perturbation: Privacy Amplification for Black-Box LLM Inference

For black-box private LLM inference, differentially private prompt privatization provides formal privacy guarantees without modifying the remote LLM, but stronger privacy typically requires heavier perturbation and degrades downstream utility. We introduce a privacy amplification framework that leverages cross-user anonymity and expands the anonymity set through controllable pseudo-prompt expansion, enabling weaker local perturbation under the same end-to-end privacy budget. To account for the resulting amplification, we develop a mechanism-aware accountant that exploits the sequential structure of prompt privatizers through likelihood-ratio moments, avoiding explicit enumeration of the exponentially large sequence output space. The accountant supports both autoregressive and token-replacement privatizers and remains tractable for sequence lengths where distribution-based mechanism-aware accounting becomes costly. Experiments on MMLU and MedicalQA show relative F1 improvements of \(8.2%\)–\(10.2%\) over the original privatization mechanisms and substantially tighter privacy amplification than generic shuffle accounting.