- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper addresses the challenge of computationally efficient preference learning under adversarially changing utilities. The authors propose Mirror-Duel, which combines symmetric pairwise comparisons with mirror descent and self-concordant regularization. They extend the method to batched comparisons and partial ranking feedback, and present regret upper and lower bounds to characterize the benefit of additional preference information. The authors also report synthetic experiments and evaluations on CartPole and Acrobot to examine regret performance and compare the proposed method with existing preference-learning approaches.
Critical strengths
-
The paper studies an important problem: learning efficiently from preference feedback when the underlying utilities change. Developing an approach that avoids maintaining confidence sets is an interesting direction for continuous decision spaces.
-
The paper considers pairwise, batched, and ranking feedback within a common framework. This provides a useful perspective on how different amounts and forms of preference information may affect learning efficiency.
Critical weaknesses
-
The main theoretical guarantees are not established by the presented arguments.
The relationships between preference regret, linear regret, and sample complexity in Remarks 1 and 2 do not hold as stated under adversarially changing utilities. The ranking-feedback reduction also contains unresolved issues concerning the comparison construction and the claimed independence of the extracted comparisons. These problems directly affect the validity of several central results.
Essential supporting details are also difficult to verify. Section 3 refers to Sections C.1 and C.3 for the pseudocode and regret analysis (lines 285–286), and to Section C for the complete proof of Theorem 1 (line 312). The batched and ranking algorithms similarly refer to Sections D.1 and E.1 for their pseudocode (lines 349 and 417). However, I could not locate these sections in the material available to me. Together with inconsistent notation and insufficient explanation in the main text, this prevents a reliable assessment of the algorithms and their theoretical guarantees.
-
The computational efficiency and practical advantages are not sufficiently established.
The paper claims an overall runtime of , but the algorithm requires Hessian eigendecomposition and a regularized optimization step at each round. It remains unclear how these operations are accounted for in the claimed complexity or whether additional assumptions are needed.
Figure 1 presents synthetic regret curves, but these do not directly establish runtime efficiency or superiority over existing methods. Section 6 refers to Sections G.3 and G.4 for the CartPole and Acrobot results and baseline descriptions (lines 475–478), but I could not locate the referenced material. Therefore, I could not verify the practical advantages claimed in the abstract or assess the fairness of the comparisons. The connection between the studied linear optimization setting and general RLHF policy optimization also remains insufficiently explained, making the practical scope of the contribution unclear.
Questions for authors
-
Under what conditions do the main theoretical reductions hold?
Please clarify the assumptions needed for the relationships in Remarks 1 and 2 to hold when utilities change over time. Relatedly, how does the ranking-feedback construction guarantee valid and independent pairwise comparisons? Which theoretical results remain valid after addressing these concerns?
-
Where can the referenced supporting material be found?
The paper repeatedly refers to supplementary sections for algorithm pseudocode, proofs, and detailed experiments, but I could not locate this material. Were these sections submitted separately? The complete algorithm descriptions and proofs are necessary to assess the proposed guarantees.
-
How is the claimed runtime obtained?
Please account for Hessian construction, eigendecomposition, and the regularized optimization step. Does this complexity require a particular decision space, barrier function, or implementation? Are there runtime measurements supporting the claimed computational advantage?
-
What evidence supports the claimed practical advantages in RLHF?
Please identify the CartPole and Acrobot results referenced in Sections G.3–G.4 and clarify the policy parameterization, preference generation, and feedback budgets used for each method. How do these experiments correspond to the theoretical setting, and what evidence supports the claimed advantages over the baselines?