acceptodds
Withdrawn from ICLR 2027

Pluralistic Off-policy Evaluation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Personalized preference alignment for LLMs with diverse human preferences requires evaluation and alignment methods that capture pluralism. Most existing preference alignment datasets are logged under policies that differ substantially from the evaluated LLMs, while existing off-policy estimators focus primarily on overall utility and ignore preference diversity. Extending off-policy evaluation to pluralistic preference alignment is non-trivial, as diversity objectives depend on the target policy distribution and cannot be reliably estimated from logged data using standard Inverse Propensity Scoring (IPS) alone. We propose a framework for offline pluralistic preference evaluation and alignment in LLMs. Our method introduces a unified reward that combines collaborative utility from human preference signals and an entropy-inspired diversity component. To enable reliable estimation under logged interactions, we derive decomposable IPS estimators that separately evaluate utility and diversity, yielding lower-variance estimates than directly applying standard IPS to the unified objective. Based on the resulting off-policy value function, our method further enables direct off-policy optimization for improved pluralistic alignment. Empirical results show that our method effectively enhances pluralistic response generation while preserving general downstream performance.

Primary area:
generative models
Keywords:
Off-policy Evaluation, Pluralistic Preference Alignment, Preference Diversity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_5AvvOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The main problem the paper is solving is off-policy evaluation and optimization of LLMs balancing both utility and diversity of a given fixed set of offline responses. The paper proposes an inverse propensity weighted reward balancing utility and diversity and presents experiment on several dataset comparing with standard SFT and DPO algorithms.

Critical strengths
  1. The motivation around off-policy evaluation of LLMs is good.
  2. IPS is notoriously fragile especially with LLM probabilities as they can blow up a lot. The paper uses various measures like length normalization, computing probabilities over a smaller choice set instead of open-ended generation; employs the standard product to sum trick to save it from vanishing; and mentions clipping and normalizing as additional measures. From the experiment it seems like these tricks are helping as IPS scores look reasonable. I also appreciate the theory around IPS sensitivity.
Critical weaknesses
  1. My main concern is regarding novelty. The theory is already known - the objective function is a standard reward + an entropy regularizer The IPS adjustments are also standard in the offline RL and recsys literature. I'm not sure what can be added to make this part novel but I'm willing to reconsider the paper if the authors can rigorously show that this method outperforms other possible baselines. For example, the current baselines with SFT and DPO are quite thin and that they will underperform is obvious. The main objective seems similar to standard RLHF objective where the KL divergence is replaced with an entropy term, can you add a baseline with RLHF + IPS score? Additionally, since theory is not very novel the experiments need to be more rigorous, can you add one or two baseline algorithms in pluralistic alignment or RLHF with diversity as an additional objective?

  2. The main selling point of the paper seems to be for pluralistic alignment. However, what is actually happening is diversity + utility optimization from a set of responses. There is a bit of mismatch in what is promised versus what is delivered. For this to be a paper on pluralistic alignment the construction of the set of responses itself needs more attention. For example if the fixed options are 90% value 1 and 10% value 2, then value 1 will be over-represented. This is a direction that can be potentially explored as a novel selling point if point 1 is not working out.

Questions for authors

I've included all my main questions and assessment in the weakness section.

Reviewer_hoNiOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes POPE, a method for offline pluralistic evaluation and alignment of LLMs. The value function combines an action local collaborative utility with a diversity reward equal to the self information of each response. Under independent with replacement draws from a known logger and slate independent feedback, both terms admit response level IPS estimators. The resulting surrogate is maximized by gradient ascent. The policy is defined as a length normalized LLM score, normalized over a finite candidate set.

Critical strengths
  • The problem is clear: offline evaluation and optimization of pluralistic behavior from logged human feedback.

  • The objective is simple and cheap to optimize.

Critical weaknesses

-The off policy framing and its guarantees do not apply to the experiments. For example, the logged responses are human-written reviews. π0\pi_0 is replaced by GPT2-Medium length-normalized likelihoods, normalized over the logged candidates.

  • The theoretical content is standard IPS unbiasedness for an additive reward under independent draws.

  • It seems that the entropy term does not need IPS at all.

  • Baselines are limited to Base, SFT, and DPO, which are designed to collapse to a single mode. There are no pluralistic or diversity-aware baselines.

  • No ablation studies.

Questions for authors

See above.

Reviewer_ZZ8vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes POPE, a framework for evaluating and training LLMs for pluralistic alignment from logged human feedback. The objective is collaborative utility (normalized feedback such as helpfulness votes) plus λ times the entropy of the policy over a fixed candidate set. The authors derive response-level IPS estimators for both terms. They show these are unbiased under independent sampling with a known logging policy (Prop. 1), and they bound the variance and the bias from proxy propensities (Prop. 2). The estimator is then used as a training objective. Experiments cover a synthetic Alpaca-GPT4 study and review generation on Reddit and Amazon data with Llama3, Qwen3 and Phi-3.5. GPT2-Medium serves as a proxy for the logging policy.

Critical strengths
  1. Combining off-policy evaluation with pluralistic alignment is timely, and the estimator is simple. The propositions are correct under their stated assumptions. The paper is also unusually honest about its limits: it notes that rank-based scores are not action-local, that entropy is not mode coverage, and that the proxy propensity carries no guarantee.

  2. SFT uses only the top response and DPO uses one best/worst pair. POPE instead uses every candidate with its feedback, which is a sensible way to use review data. On Phi-3.5, POPE is best or near-best on almost every metric across all four datasets.

Critical weaknesses
  1. The objective does not match the goal, and the theory does not cover the experiments. Response entropy is a weak proxy for pluralism; the authors admit this, and Fig. 2 even shows a lower-entropy policy covering more modes. With η ∈ [0,1] and λ = 1, the optimal policy is π ∝ exp(u/λ), so the best and worst candidates differ in probability by at most a factor of e. In effect, the objective pulls the policy toward the uniform distribution and moves mass onto low-quality reviews. Because the utility term is itself a majority vote, all pluralism comes from the entropy term. In the real experiments the logging policy is a GPT2 proxy with no error bound, so Props. 1–2 do not apply. The results reflect these issues: on Llama3, POPE loses helpfulness and relevance to Base and DPO (e.g., Amazon Movies relevance 48.38 vs. 57.71 and 57.57), and on Qwen3 Amazon Movies its relevance is about 10 points below DPO. Claims such as “consistently improves” and “robust, domain-agnostic” are therefore too strong. The authors could compute entropy only over acceptable responses or over modes, or directly optimize the coverage measure C_K the paper already defines.

  2. The evaluation does not measure pluralism well, and key ablations are missing. Distributional Alignment is just the normalized entropy of the induced distribution, with no target, so outputs unrelated to every reference produce a uniform distribution and get the maximum score of 1. The vote-based θ_k is available but unused, and JSD(p, θ) would be a real alignment metric. Coverage counts individual references rather than opinion categories, and cosine similarity at δ = 0.8 mostly reflects topic, not stance. V̂(θ) is the training objective computed with the same proxy, so POPE winning on it is expected. The lexical metrics also disagree: on Amazon Movies, SFT has the lowest Distinct-N in Table 2, yet in Fig. 7b it appears to have the best Self-BLEU on Llama and Qwen, where POPE ranks worst on Llama. Fig. 5 omits Base, Qwen3, and error bars, and the table captions claim a blanket p < 0.05 even where POPE is not best. No ablation separates the effects of λ, the IPS weights, and using K responses instead of one pair. Comparing the sentiment of generated reviews with star-rating distributions, measuring aspect coverage, clustering references into modes, and adding a small human study would make the evaluation much more convincing.

Questions for authors
  1. Where does η come from in the Alpaca-GPT4 study, since the GPT-4o references have no votes? How are votes normalized per item in the review data?

  2. How was λ = 1 chosen, and by which metric? Please report a λ sweep (including λ = 0) on helpfulness, relevance and coverage.

  3. What do the IPS weights from a GPT2 proxy add? Please compare against POPE with all weights set to 1, and against DPO trained on all pairs (or the Plackett–Luce version of DPO).

  4. Why does Fig. 6 have no “baseline wins” category? Were losses merged into ties? Please give the judge prompt and say whether the order of the two response sets was randomized.

  5. How are Distinct-N and Coverage computed: within one query or over the whole test set, with how many samples per query, at what temperature, and with which embedding model? How sensitive is Coverage to δ?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.