- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper considers the contextual duelling bandit problem which targets optimising action selection from qualitative human feedback signals in a form of pairwise comparisons. They follow the standard Bradley-Terry assumption in the literature which states that the comparisons are dictated by a latent utility function. The main claim of the paper is that traditional approaches such as MaxInP (Saha, 2021) explores aggressively which can lead to queries to a user between two bad choices. Instead the author propose a constrained problem where at least one of the two alternatives presented to the user is within fraction of a known baseline utility value. The authors consider the problem under the assumption of linear utilities and more broader general class of functions and propose two algorithms, MaxInP-BC and BT-IGW-BC respectively, that can safely explore while achieving sublinear regrets. The two algorithms are extending unconstrained counterparts and numerical experiments show that they deliver on their promise of safely exploring the action space at some cost compared to the unconstrained algorithms.
Critical strengths
- The paper provide a reasonable regret analysis and theoretical guaranteees for the proposed algorithms
- The paper is well-scoped and easy to follow. Clear what are the problems the authors address and how.
- The paper goes beyond the linear utility assumption to provide a broader and more applicable theoretical analysis.
Critical weaknesses
- In section 2, the authors call their problem formulation contextual duelling bandits, however, I believe this setting is the continuous duelling bandit problem and not the contextual. In their definition, the arms are themselves are a set from a continuous -dimensional space, while in the standard definition by Dudik et al. (2015), the context is an auxiliary information that the bandit observes before making a selection affecting all candidate actions simultaneously.
- The paper lacks sufficient literature review and comparison. The authors only build on selected previous publications without addressing many more recent results of the field and do not provide sufficient comparison to previous works. One of the main claims of the paper is that duelling bandit algorithms are exploring too aggressively. While this is the case for MaxInP which relies purely on exploration, other algorithms such as POP-BO (Xu et al. 2024), MaxMinLCB (Pasztor et al., 2024), or BOHF (Kayal et al., 2025) this is less evident.
- The main motivation is to gather human feedback without showing only low quality options to users which would motivate a safety constraint that holds for each round of actions separately instead the authors propose a cumulative safety constraint in definition 2.1.
- The experimental results lack evidence of generalization. The paper does not describe the definition of the utility functions furthermore it tests the linear and general settings both only on a single utility function that does not provide sufficient evidence that their results generalize across the whole function space.
Questions for authors
- How do the regret bounds compare to previous works in duelling bandits and what is the additional factor due to the constraints?
- How do experimental results generalize to other utility functions especially in the general case, furthermore, have the authors validated performance on real world datasets to support their motivation in the introduction?