- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes Neural-ADB, an active contextual dueling-bandit algorithm for efficiently collecting human preference feedback under nonlinear reward functions. The approach combines uncertainty-based context selection, neural reward estimation, and UCB- or Thompson-sampling-based arm selection. Under the Bradley–Terry–Luce preference model, the authors derive theoretical bounds on the worst-case suboptimality gap of the learned policy and claim a sublinear convergence rate as the preference dataset grows. The proposed framework extends existing active contextual dueling-bandit methods beyond linear reward models.
Critical strengths
-
The paper addresses an important problem of reducing the number of human preference observations required for effective preference learning.
-
The work relaxes the linear reward assumption commonly adopted in prior active preference-learning methods, making the framework potentially more applicable to practical settings with nonlinear reward functions.
-
The Qwen3-4B experiment provides preliminary evidence that the proposed acquisition strategy can be incorporated into an LLM preference-optimization workflow.
-
The experimental section includes several useful ablations, examining the effects of varying the number of arms, feature dimensions, and neural network architectures.
Critical weaknesses
-
The context selection and neural arm selection strategies are largely adapted from prior work (Das et al., 2024; Verma et al., 2025). While their combination is interesting, the authors should more clearly articulate the novel algorithmic and theoretical contributions beyond these existing methods.
-
In Appendix B.1, the authors appear to incorrectly equate the conditional expectation of the accumulated Gram matrix with the sum of one-step conditional expectations. Consequently, the application of Matrix Azuma does not directly justify the stated bound. Could the authors clarify or correct this step in the proof?
-
The claimed convergence rate depends on the collected data providing sufficient exploration across the feature space. However, the authors do not establish that the proposed context and arm selection strategies guarantee this coverage. Consequently, the claimed sample-efficiency guarantee is not fully supported by the theoretical analysis.
Questions for authors
-
The claimed convergence rate relies on sufficient exploration of the feature space. Can the authors theoretically establish that Neural-ADB’s context and arm selection strategies guarantee this condition, rather than assuming it holds?
-
In Appendix B.1, the conditional expectation of the accumulated Gram matrix appears to be incorrectly equated with the sum of one-step conditional expectations. Can the authors clarify this step and provide a rigorous justification for the application of Matrix Azuma?