- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This work approaches the problem of active acquisition of preference data for learning verifiers. It raises the concern that most methods in the literature focus on reducing epistemic uncertainty in the verifiers without awareness of the decision operator itself. It then proposes DFA, a method that targets the posterior variance of a linearised decision loss, rather than the posterior over the weights. The paper provides experimental evidence in several datasets (preference learning, coding, tool use), showing that the proposed method outperforms other acquisition criteria and one active learning recent method (ActiveDPO) in single batch acquisition.
Critical strengths
-
The paper is well written and well presented. Clear to understand. It also approaches a relevant problem (active learning for verifiers).
-
The paper presents a good experimental design: it asks relevant questions, provide evidence in a good number of benchmarks. The results/plots are clear and communicate the claims.
-
The employed methodology is novel, and the derivation is solid and mathematically well justified. Furthermore, the algorithm and practical implementation is mindful about scalability and discusses important aspects of the algorithm complexity costs.
Critical weaknesses
-
The work build on top of the motivation that many previous methods are based on parameter (epistemic) uncertainty as the criteria to select samples, but misses standard baselines in the literature like BALD [1] and variants [2], which were also used in the past in the same problem setting [3].
- In this same line, there are some recent work that is very relevant to this work but missing in the related work section [3-7]
-
The work adopts a particular posterior approximation based on linearization and develops a decision criteria on top. However, other forms of posteriors are widely used in literature, from GPs to Deep Ensembles (which can be also easily approximated on top of frozen features). The work does not discuss the particular design choice for this posterior approximation, nor compare with other possibilities. It is also unclear how much the linearity assumptions affect the posterior approximation and the downstream decision.
-
While the experimental setup is very clear and convincing, it looks simplistic: a single batch acquisition, with a small batch of 32 samples. Usually preference learning deals with very large batches, iteratively. This is a crucial design because batch active learning is a challenge on its own (as opposed to single-point acquisition), and the work does not discuss this.
-
No discussion on limitations of the method, which I believe is important for the reader.
References
[1] Lindley. On a measure of the information provided by an experiment. The Annals of Mathematical Statistics, 1956.
[2] Kirsch et. al. Stochastic Batch Acquisition: A Simple Baseline for Deep Active Learning, 2021.
[3] Melo et. al. Deep Bayesian Active Learning for Preference Modeling in LLMs, 2023.
[4] Gleave and Irving. Uncertainty estimation for language reward models, 2022.
[5] Mehta et al. Sample efficient reinforcement learning from human feedback via active exploration, 2024.
[6] Ji et al. Reinforcement learning from human feedback with active queries, 2024.
[7] Dwaracherla et al. Efficient exploration for LLMs, 2024.
Questions for authors
- Is the experimental setup only about a single-batch acquisition? Are there any results on sequential acquisition?
2 comments
Dear authors,
Thank you for answering and commenting on the raised points and questions. It helped clarifying my questions, although no new evidence was provided in order to change my perspective in the paper. Therefore, I will maintain my score.
Response to Reviewer 3
We thank the reviewer for the positive assessment of the paper and for recognizing the strength of the experimental design and methodology. We respond to the questions below.
1. Sequential acquisition
In all experiments, we perform sequential acquisition in batches of eight labels. After each batch, we refit the verifier and recompute its posterior using all labels acquired so far. Within a batch, each selection is also reflected in the criterion through a rank-one covariance update before the next pick, helping avoid redundant comparisons. These alternating steps are detailed in Algorithm 1 and Section 3.3.
2. Epistemic-uncertainty baselines
We thank the reviewer for pointing us to this relevant literature. We will add these references and discuss how their epistemic-uncertainty criteria differ from targeting the deployed decision.
3. Choice of posterior approximation
We agree that other posterior approximations could be used in the decision criterion. We chose Laplace because the main experiments use a regularized convex Bradley–Terry verifier, for which a local Gaussian approximation is natural and computationally practical (Bishop (2006)). Deep ensembles would require fitting multiple models and add computation cost.
To assess the Laplace approximation, we used Monte Carlo samples from the Bradley–Terry posterior as a reference on all four datasets at B=32. We compared the two posteriors along the decision direction. The Laplace variance differs from the sample-based estimate by 0–9% on average, and the resulting acquisition rankings from the true posterior agree with Laplace’s rankings at Spearman correlation 0.95–0.99. Finally, the top candidate selected using the two posterior estimates agrees on 92–98% of evaluation prompts. These results support using Laplace in our linear-verifier setting, and we will include the sampling procedure and comparisons in the revision.
We will also add limitations of the current work in the final draft.
Christopher M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006.