Pluralistic Off-policy Evaluation

Personalized preference alignment for LLMs with diverse human preferences requires evaluation and alignment methods that capture pluralism. Most existing preference alignment datasets are logged under policies that differ substantially from the evaluated LLMs, while existing off-policy estimators focus primarily on overall utility and ignore preference diversity. Extending off-policy evaluation to pluralistic preference alignment is non-trivial, as diversity objectives depend on the target policy distribution and cannot be reliably estimated from logged data using standard Inverse Propensity Scoring (IPS) alone. We propose a framework for offline pluralistic preference evaluation and alignment in LLMs. Our method introduces a unified reward that combines collaborative utility from human preference signals and an entropy-inspired diversity component. To enable reliable estimation under logged interactions, we derive decomposable IPS estimators that separately evaluate utility and diversity, yielding lower-variance estimates than directly applying standard IPS to the unified objective. Based on the resulting off-policy value function, our method further enables direct off-policy optimization for improved pluralistic alignment. Empirical results show that our method effectively enhances pluralistic response generation while preserving general downstream performance.