- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The main problem the paper is solving is off-policy evaluation and optimization of LLMs balancing both utility and diversity of a given fixed set of offline responses. The paper proposes an inverse propensity weighted reward balancing utility and diversity and presents experiment on several dataset comparing with standard SFT and DPO algorithms.
Critical strengths
- The motivation around off-policy evaluation of LLMs is good.
- IPS is notoriously fragile especially with LLM probabilities as they can blow up a lot. The paper uses various measures like length normalization, computing probabilities over a smaller choice set instead of open-ended generation; employs the standard product to sum trick to save it from vanishing; and mentions clipping and normalizing as additional measures. From the experiment it seems like these tricks are helping as IPS scores look reasonable. I also appreciate the theory around IPS sensitivity.
Critical weaknesses
-
My main concern is regarding novelty. The theory is already known - the objective function is a standard reward + an entropy regularizer The IPS adjustments are also standard in the offline RL and recsys literature. I'm not sure what can be added to make this part novel but I'm willing to reconsider the paper if the authors can rigorously show that this method outperforms other possible baselines. For example, the current baselines with SFT and DPO are quite thin and that they will underperform is obvious. The main objective seems similar to standard RLHF objective where the KL divergence is replaced with an entropy term, can you add a baseline with RLHF + IPS score? Additionally, since theory is not very novel the experiments need to be more rigorous, can you add one or two baseline algorithms in pluralistic alignment or RLHF with diversity as an additional objective?
-
The main selling point of the paper seems to be for pluralistic alignment. However, what is actually happening is diversity + utility optimization from a set of responses. There is a bit of mismatch in what is promised versus what is delivered. For this to be a paper on pluralistic alignment the construction of the set of responses itself needs more attention. For example if the fixed options are 90% value 1 and 10% value 2, then value 1 will be over-represented. This is a direction that can be potentially explored as a novel selling point if point 1 is not working out.
Questions for authors
I've included all my main questions and assessment in the weakness section.