acceptodds
Under review as a conference paper at ICLR 2027

Provably Sample-Efficient Active Preference Data Collection

Abstract

Collecting human preference feedback is often expensive, leading recent works to develop algorithms to select them more efficiently. However, these works assume that the underlying reward function is linear, an assumption that does not hold in many real-life applications, e.g., online recommendation. To address this limitation, we propose Neural-ADB, an algorithm based on the neural contextual dueling bandit framework that provides a practical method for collecting human preference feedback when the underlying latent reward function is non-linear. We theoretically show that when preference feedback follows the Bradley-Terry-Luce model, the worst sub-optimality gap of the policy learned by Neural-ADB decreases at a sub-linear rate as the preference dataset increases. Our experimental results on preference datasets further corroborate the effectiveness of Neural-ADB.

Primary area:
learning theory
Keywords:
Active Learning, Human Feedback, Preference Feedback, Neural Dueling Bandit, Contextual Dueling Bandit

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_C1THOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Neural-ADB, an active contextual dueling-bandit algorithm for efficiently collecting human preference feedback under nonlinear reward functions. The approach combines uncertainty-based context selection, neural reward estimation, and UCB- or Thompson-sampling-based arm selection. Under the Bradley–Terry–Luce preference model, the authors derive theoretical bounds on the worst-case suboptimality gap of the learned policy and claim a sublinear convergence rate as the preference dataset grows. The proposed framework extends existing active contextual dueling-bandit methods beyond linear reward models.

Critical strengths
  1. The paper addresses an important problem of reducing the number of human preference observations required for effective preference learning.

  2. The work relaxes the linear reward assumption commonly adopted in prior active preference-learning methods, making the framework potentially more applicable to practical settings with nonlinear reward functions.

  3. The Qwen3-4B experiment provides preliminary evidence that the proposed acquisition strategy can be incorporated into an LLM preference-optimization workflow.

  4. The experimental section includes several useful ablations, examining the effects of varying the number of arms, feature dimensions, and neural network architectures.

Critical weaknesses
  1. The context selection and neural arm selection strategies are largely adapted from prior work (Das et al., 2024; Verma et al., 2025). While their combination is interesting, the authors should more clearly articulate the novel algorithmic and theoretical contributions beyond these existing methods.

  2. In Appendix B.1, the authors appear to incorrectly equate the conditional expectation of the accumulated Gram matrix with the sum of one-step conditional expectations. Consequently, the application of Matrix Azuma does not directly justify the stated bound. Could the authors clarify or correct this step in the proof?

  3. The claimed convergence rate depends on the collected data providing sufficient exploration across the feature space. However, the authors do not establish that the proposed context and arm selection strategies guarantee this coverage. Consequently, the claimed sample-efficiency guarantee is not fully supported by the theoretical analysis.

Questions for authors
  1. The claimed convergence rate relies on sufficient exploration of the feature space. Can the authors theoretically establish that Neural-ADB’s context and arm selection strategies guarantee this condition, rather than assuming it holds?

  2. In Appendix B.1, the conditional expectation of the accumulated Gram matrix appears to be incorrectly equated with the sum of one-step conditional expectations. Can the authors clarify this step and provide a rigorous justification for the application of Matrix Azuma?

Reviewer_iGvFOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies active collection of pairwise preference data, framed as a contextual dueling bandit with a nonlinear latent reward and a logistic (BTL) preference link. The proposed method, Neural-ADB, works in three steps:

It picks the context whose arm pairs have the largest uncertainty in gradient features (Eq. 3, adapted from APO). It picks a greedy first arm and an optimistic (UCB) or Thompson-sampled second arm (Eqs. 4, 5, 7, adapted from neural dueling bandits). It fits the reward with a bias-free ReLU network in the NTK regime. The theory has three parts:

Lemma 1 lower-bounds λ_min of the adaptively built Gram matrix using matrix Azuma. Lemma 2 reduces worst-case suboptimality to a uniform confidence width. Theorems 1–2 claim a worst-case suboptimality gap of Õ(√(d̃/T)), and a (log T)^½ improvement over prior active preference methods. Experiments use two synthetic rewards, 10(xᵀθ)² and 2sin(xᵀθ), with d ≤ 40 and K ≤ 20, plus one LLM alignment run on TL;DR with Qwen3-4B, where the same reward model labels the data and judges the outputs.

Critical strengths

The problem is relevant, and the paper correctly notes that cumulative-regret bounds don't imply uniform suboptimality bounds. The analysis is modular and candid. Lemma 2 is a clean reduction, and the paper openly states the eigenvalue assumption (lines 926–927, 998–1000). The synthetic experiments are controlled: 10 repetitions with 95% CIs, sweeps over d, K, and width, and matched neural baselines.

Critical weaknesses

The rate doesn't follow. Two independent reasons: (a) The network is bias-free, so scaling W_l by eˢ and W_{l+1} by e⁻ˢ leaves the output unchanged. The direction (…, vec W_l, −vec W_{l+1}, …) at θ₀ is therefore orthogonal to every gradient feature, so λ_min(Σ̄_T) = 0 for all T and Lemma 1 is always trivial. (b) Separately, λ_min ≤ L²/p, so the nontrivial branch needs T > 8p²·log(p/δ). For the paper's own network, p ≈ 3,550, that is about 10⁹ rounds, against T = 1,000. This undercuts the claimed rate; it doesn't show the algorithm fails. The algorithm isn't connected to coverage. Neither Eq. 3 nor the arm pair actually queried appears in the proofs, so the rate isn't established for this algorithm, and neither is the log-factor improvement under comparable assumptions. Impossible synthetic values. With f = 2sin(·), gaps are at most 4 and cumulative regret at 300 rounds is at most 1,200. Fig. 1(d–f) shows gaps around 20, MAE around 13, and regret around 2,500–2,700. The curves also stop at 300 rounds, though the text sets T = 1,000. Realizability isn't justified. The bias-free gradient features are degree-1 homogeneous in the input, but the square and sine rewards aren't, so exact realizability over the feature cube is impossible. The paper also doesn't explain why the imported confidence bounds hold across all contexts at once. Weak evidence for the arm-selection rule. In Fig. 1(a), the APO ablation matches or beats UCB on the paper's own objective, and TS is worse. There is no passive neural baseline. The LLM experiment is unclear. There are no replication details or uncertainty estimates, and the evaluator coupling the paper acknowledges is never resolved. Fixable proof errors. The conditional-expectation identity at lines 823–825 is wrong; it can be repaired with Azuma on Σ(z_s z_sᵀ − Σ_s). The 1/w normalization doesn't match between the two Gram matrices. λ₀ is never defined.

Questions for authors

Do you agree that the bias-free homogeneity gives λ_min(Σ̄_T) = 0? Under what parameterization can it be bounded away from zero? How is the nontrivial branch of Lemma 1 compatible with w ≥ poly(T)? Please state the polynomial. Can you link Eq. 3 and the arm pair actually queried to coverage? Is the reward realizable on the whole cube or only on a finite pool? Why do the confidence bounds cover all contexts? How do the sine plots exceed 4 for the gaps and 1,200 for regret? Why do they stop at 300 rounds? Can you add a passive baseline, and explain how the Random baseline produces a policy? How many runs underlie Fig. 7, with what uncertainty? Can you use an independent judge?

Reviewer_eDhJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper tackles the learning of contextual dueling bandits where the latent reward is nonlinear. To solve this problem, the proposed algorithm combines to existing components from the literature in contextual dueling bandits: an algorithm framework and a UCB trick for arm selection. The authors provide theoretical characterization of the proposed algorithm, and evaluated the algorithm synthetic benchmarks.

Critical strengths

The proposed algorithm extends previous methods to the setting with nonlinear latent reward functions. The algorithm design is modular and clean. Both theoretical and empirical evaluations of the algorihtm are provided.

Critical weaknesses

Below are the weaknesses that seem critical to the reviewer. The reviewer is willing to raise the score if they can be properly addressed.

  1. The theoretical result is not fully meaningful. The theoretical guarantee of the proposed algorithm is characterized by term λmin⁡(ΣˉT)\lambda_{\min}(\bar{\Sigma}_T) in the denominator. This term can be arbitrarily small, making the error bound arbitrarily. It is a non-trivial lower bound of this term that shows the impact of active arm selection. Bounding this term is also the core difficulty to most papers in similar settings (linear bandits, dueling bandits, etc.). However, the theoretical guarantees of this paper does not characterize this critical term.

  2. Lower-bounding the critical term of λmin⁡(ΣˉT)∈Rp×p\lambda_{\min}(\bar{\Sigma}_T)\in \mathbb{R}^{p\times p} does not seem feasible in the setting of this paper. Assume this matrix is full rank. This implies that rank≥prank \geq p, since this matrix is p×pp\times p. However, the matrix must have rank smaller than TT since it is a summation of T rank-1 matrices, leading to T≥rank≥pT \geq rank \geq p. This contradicts the assumption required by Lemma 3, p≥poly(T)p \geq poly(T). Therefore, under the setting and assumptions of this paper, there does not seem to be a non-trivial lower bound of this matrix, leading to an arbitrarily large error bound in Theorem 1 and 2.

  3. The problem setting appears to be an special case of the settign of APO-gen in [Das, 2024]. The advantage of the proposed algorithm compared to APO-Gen is not clear.

  4. The algorithm design is incremental. The proposed algorithm, Neural-ADB, combines three components from existing literature: the algorithm framework and context-selection from [Das, 2024], adapted to the nonlinear setting by replacing the gradient of the linear reward to the gradient of the nonlinear neural network, and the arm-pair rule of [Verma, 2025].

Questions for authors

Please see the "critical weaknesses" section for critical questions that materially affect your assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.