- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies the problem of constructing KL-regularized response policies directly from pairwise comparisons, assuming a fixed finite set of candidate responses for each prompt. The authors propose an MCMC-based sampling method that constructs a Markov chain whose stationary distribution corresponds to the desired KL-regularized policy under the Bradley–Terry preference model. The paper further analyzes the finite-sample error of the proposed sampling procedure and develops an adaptive feedback allocation strategy across multiple prompts based on their sampling difficulties. The authors provide theoretical guarantees for the proposed approach and evaluate its effectiveness through synthetic experiments and an LLM-based question-answering task.
Critical strengths
-
The paper presents an elegant approach to constructing KL-regularized policies through MCMC sampling. The idea of designing a Markov chain that converges to the desired policy distribution using only pairwise comparisons is clever and technically interesting. This provides an alternative to explicitly learning reward models or parameterized policies.
-
The proposed approach is supported by a relatively comprehensive theoretical analysis. Beyond designing the MCMC sampling procedure, the authors characterize its finite-sample approximation error and derive a principled feedback allocation strategy across multiple prompts. These results provide theoretical support for the proposed method and offer insights into its sampling efficiency.
-
The experimental results on synthetic datasets are promising. The proposed adaptive sampling strategy achieves performance close to the oracle allocation with known sampling difficulties and consistently outperforms simpler allocation baselines. These results provide empirical support for the effectiveness of the proposed approach.
Critical weaknesses
-
The proposed method is restricted to a fixed finite set of candidate responses, which limits its applicability to modern LLM generation scenarios. In practice, LLMs typically generate responses autoregressively over an extremely large output space, rather than selecting from a predefined discrete candidate set. This creates a considerable gap between the proposed formulation and the predominant generation paradigm, raising concerns about the broader applicability of the method.
-
The practical motivation for the specific problem setting is not sufficiently justified. In realistic applications, it is unclear how frequently one would encounter a scenario where a fixed finite set of candidate responses is available and the objective is specifically to sample from a KL-regularized policy over these candidates. Even when such a candidate set exists, directly selecting or ranking high-quality responses may be more natural than sampling from a prescribed distribution. The authors should provide stronger justification for the necessity and practical relevance of this problem formulation.
Questions for authors
Please see the weaknesses.
Thanks for the rebuttal, which has addressed my concerns. I am happy to maintain the positive score