- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Population-Relative RLHF, a framework for improving the quality-diversity trade-off in LLM alignment by incorporating pairwise preference feedback conditioned on a reference population of responses. The problem is formulated as a bi-level optimization task, where the upper level learns a population-relative reward model and the lower level optimizes the LLM policy through a mean-field-type game. They develop a first-order, Hessian-free algorithm (FD-S-MFG) with finite-time convergence guarantees, alongside a simplified, memory-efficient variant for practical LLM training. Experiments shows that the proposed methods achieve a good quality-diversity Pareto frontier compared to the baselines.
Critical strengths
- The paper develops a novel first-order, Hessian-free algorithm with finite-time convergence guarantees for a general class of bi-level optimization problems involving mean-field-type games.
- The new method achieves a superior trade-off between quality and diversity over several competitive baselines, while its simplified variant demonstrates comparable performance with lower computational and memory overhead.
Critical weaknesses
- Limited experimental datasets. The experimental evaluation is limited to two benchmarks, Happy Tweet Generation and LitBench. The relatively narrow selection of datasets raises concerns about the generalizability and robustness of the proposed method. Evaluating the approach on additional datasets would help provide stronger empirical evidence for its effectiveness.
- Lack of computational efficiency analysis. The paper lacks quantitative comparisons of computational costs against existing baselines. Reporting training time, GPU memory usage, and computational requirements would help assess the practical efficiency and scalability of the proposed approach.
- Limited justification of theoretical assumptions. The convergence guarantee in Theorem 1 relies on three key assumptions. Although the authors briefly discuss these assumptions and cite related literature, they provide limited justification for their validity in practical LLM alignment settings. A more detailed discussion of their applicability would strengthen the theoretical contribution.
Questions for authors
- Could the authors empirically validate the convergence rate and sample complexity bound established in Theorem 1, perhaps through some controlled tabular experiments?
- Could the authors clarify the motivation for selecting Jaccard distance and Vendi score as diversity metrics? Additionally, evaluating the proposed method using alternative diversity metrics would help establish whether the improvements generalize beyond the metrics.
- I think there is a typo in the point annotations of Figure 1 and 2. It should be instead of .