- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies active preference learning under a realizable linear logistic model. The goal is to identify the preference direction of every response pair in a finite pool using as few labels as possible. The authors propose a margin-aware experimental design algorithm and analyze its instance-dependent label complexity. They also compare the leading complexity term with a KL-based lower bound under bounded logits. A simpler greedy method selects pairs according to their confidence radius minus their estimated absolute margin. Experiments evaluate the methods on synthetic data and human preference data, including preference classification and best-of- response selection.
Critical strengths
-
The sampling objective is well motivated by the classification goal. A pair with a wide confidence interval may already have a clear preference direction if the interval excludes zero. Accounting for the interval’s location gives a concrete reason to move beyond uncertainty reduction alone.
-
The instance-dependent analysis is useful. The complexity measure captures both the geometry of the feature set and the difficulty of individual comparisons through their margins and Fisher information. The comparison with the KL lower bound helps explain what this measure represents and under which conditions its dependence on the instance is near optimal.
-
The greedy rule has a clear interpretation and is straightforward to understand. The experiments also consider both held-out preference accuracy and response selection, providing evidence about its usefulness beyond the theoretical identification objective.
Critical weaknesses
-
My most important concern is that the main text should include the key experimental results. All experiments are currently deferred to the appendix, leaving little empirical evidence in the main paper. For a paper motivated by label-efficient RLHF, these results are important for assessing the practical value of the proposed methods. The main text should present the basic experimental setup and representative results on preference classification and best-of- selection. Additional implementation details and supplementary results can remain in the appendix.
-
The contributions and theoretical results need a clearer presentation. The paper introduces an oracle allocation, a phased experimental design algorithm, and a practical greedy method, but their relationship and respective contributions are not explained clearly enough. The introduction would benefit from a concise account of the main theoretical guarantees, their assumptions, and the precise sense in which the results are instance dependent and near optimal. A clearer explanation of how each algorithm follows from the central intuition would also help readers understand the technical development.
-
The novelty relative to prior work needs to be articulated more precisely. The paper discusses existing experimental design methods, but the comparison should go beyond differences in their objectives. It should explain which algorithmic components and theoretical arguments are new, which build on existing frameworks, and what additional guarantees are obtained. In particular, a more direct comparison with related work on preference learning and margin-aware pure exploration would help distinguish advances in the problem formulation, sampling strategy, and analysis.
Questions for authors
Could the authors include the key experimental details and results in the main text? Could they also clarify the main algorithmic and theoretical contributions, explain the intuition behind the proposed methods, and provide a more explicit comparison with prior work, as discussed in the Weaknesses section?
4 comments
Thank the authors for responding and clarifying. The instance-dependent theoretical results in this paper are interesting to me. Yet I still believe that this paper would benefit much from improving the presentation, especially in the Introduction and Experiments. I strongly encourage the authors to upload a revised version to address the mentioned points to make the work better prepared.
Thank you again for recognizing the value of our instance-dependent theoretical results. We have uploaded a revised one, with particular attention to the following: (1) Introduction: we clarified our contributions and the roadmap from the central intuition to the proposed design, including the connections between the oracle allocation and the two algorithms. (2) Related work: we highlighted the differences between our work and prior work, particularly in the upper-bound guarantees, algorithm adaptivity, and use of confidence intervals. (3) Experiments: we added representative experimental results to the main text.
Thanks the authors for responding. My major concerns have been resolved with the revised version. I would like to keep my positive score.
We thank the reviewer for finding our work well motivated and the instance-dependent analysis useful! Most of the concerns and questions are about presentation, we will address them carefully in the revision.
Experiments.
Thank you for the suggestion. We agree that placing all experimental results in the appendix makes the presentation less ideal. We prioritized the theoretical contributions in the main text. In the revision, we will move the highlighted experimental results to the main text for better visibility.
Clarity.
We will address these clarity issues and make the connection between the oracle, Algorithm 2, and Algorithm 3 more explicit in the paper. In summary, for the algorithmic side contribution,the intuition in Section 3.2 is the starting point: the oracle formalizes the ideal objective as if the true instance were known, Algorithm 2 aims to achieve the optimal leading instance-dependent sample complexity even when the true instance is unknown, and Algorithm 3 provides a more practical approach. For the theoretical side contribution, all existing results we are aware of are worst case only and do not capture instance-dependent behavior, whereas our analysis provides instance-dependent guarantees. We will also add a more detailed comparison with prior work in the related work section, including the comparison perspectives provided by Reviewer 3aMM.