- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Standard LLM alignment learns relative preferences between candidate responses, which can fail when both candidates are not acceptable. This paper proposes a method called DUAL aiming to address this by maintaining a joint online Gaussian belief over a user's feature preferences and a latent acceptance threshold to support a three-way decision space, consisting of Serve A, Serve B, or Abstain. Under a constrained query budget, their method uses local Bayesian D-optimality to pick between pairwise and absolute acceptability questions.
Critical strengths
-
The formulation of acceptance of abstention in LLM serving is interesting and to my knowledge, novel. The idea that preferring a response over another doesn’t mean the preferred response is acceptable is intuitive and correct and is indeed an issue with the standard pairwise preference alignment. The proposed framework Is also intuitive and interesting. It combines an acceptance threshold, abstention, and two feedback channels in one framework. The theoretical analysis provided for the unmodified recursion is also solid. In particular, the almost-true convergence for the original online recursion under adaptive query selection and continuing answer quotas is sound and strong.
-
The proposed framework is lightweight and efficient. The DUAL method uses one shared belief for serving, question selection, and timing. The information score also has a closed form from the determinant lemma, so the cost per step is low.
-
The experimental report benefits from an exceptional rigour and transparency which I appreciate. The authors clearly distinguish development exploration from confirmatory evaluations, apply strict Bonferroni adjustments to multi-contrast hypothesis families, and most importantly, openly report unresolved and null results instead of over claiming about the empirical wins. This rigorous analysis enhances the integrity and trustworthiness of the work and its findings. The controlled ablations that isolate the contributions (Table 1 and 2) are particularly beneficial.
Critical weaknesses
-
While I appreciate the transparency and integrity in the reported results, my understanding from the reported results is that when tested on actual human ratings in PRISM with a population-initialized prior, the serving AUC difference between mixed feedback which is 58.14% and absolute-only feedback 57.86% with an overlapping CI is statistically unresolved. It seems that the majority of the performance comes from the population prior and not the personal feedback, so the empirical justification of dual-channel elicitation is unclear to me.
-
The simulation does not appear to be conducted in the most accurate way. The users are generated from the same linear-logistic model that the learner assumed. Acceptance threshold are quantities of the same utility, and answers are drawn from Eq 3 and Eq 4 at unit temperature. The learner is thus correctly specified by design. The misspecification test also confirms the fragility. A hidden quadratic preference component reverses the timing advantage, and recalibration doesn’t salvage it entirely. In practical applications, real users have non-linear, context-dependent, with different levels of noise, and inconsistent preferences. Applying the same fixed noise scale is also an inappropriate decision that favours the information score’s assumptions.
-
The theorem 3 has strict assumptions that don’t seem realistic to me. The candidate features are uniform over [-1,1]^d, a correct specified linear logistic user model is assumed, and true logits bounded by 1.76.In reality, response embeddings are rarely uniform, and real-world preferences may have non-linearities and extreme logits when there are near-deterministic choices.
Questions for authors
Please find my comments in the weaknesses part. In addition to those, Have you conducted any analysis on the sensitivity of the AUC gains to the assumed noise temperature, which is fixed at 1 for both channels?
3 comments
I appreciate the authors' rebuttal and maintain my positive assessment of the work. I thank the authors for having been transparent and thorough in adding ablations and caveats, rather than overclaiming.
Thank you for your feedback
Dear Reviewer Kq5g,
Thank you for taking the time to read our response and for maintaining your positive assessment. We’re glad the added analyses and clarifications were helpful. Your feedback helped us improve the paper and make both the results and their limitations clearer.
Response to Reviewer Kq5g
Thank you for recognizing the formulation, lightweight computation and convergence analysis. Your concerns motivated clearer population controls, new feedback-noise tests and broader sufficient conditions. These additions are marked in blue.
W1. Population initialization and the evidence for mixing
The population prior is a strong contributor on PRISM. On the same 90-participant cohort, population-only, mixed and acceptability-only models achieve 58.89%, 58.14% and 57.86% serving-mistake AUC. Mixed minus population-only is -0.75 points [-3.32, +1.81], and mixed minus acceptability-only is 0.28 [-0.50, +1.06], using exploratory marginal 95% intervals. Appendix G.6 places these controls beside the updating methods and reports trivial-policy comparisons.
We frame PRISM as a component evaluation using recorded ratings and a fixed acceptability threshold. The stronger evidence for adaptive mixing comes from the prospectively frozen simulation: DUAL achieves 14.71% versus 15.94% for adaptive acceptability-only on 200 fresh profiles, with adjusted difference -1.229 points [-1.866, -0.592]. Section 6.4, Table 3 and Appendix G.2 report that comparison.
W2 and noise question. Unequal and changing feedback reliability
Feedback reliability changes which channel is worth buying. We varied true acceptability/pairwise temperatures across (1,1), (1,2) and (2,1) while keeping the learner at (1,1). Temperature 2 makes answers less decisive. Each policy was separately calibrated, then assessed on the same 120 fresh profiles with a 940-unit cap over 9,402 interactions.
At equal prices, noisier acceptability feedback gives a DUAL-minus-acceptability-only AUC difference of -4.465 points [-5.704, -3.274]. Noisier pairwise feedback instead favors acceptability-only by 1.326 points [+0.292, +2.360]. With noisier acceptability feedback, the difference remains -3.168 points at twice the pairwise price. Table 50 reports the full 15-comparison adjusted family for the original score with cost-cap timing.
Appendix G.5 also tests unannounced noise changes without recalibration. The learner has no reliability parameter. These tests address feedback mismatch, while the retained hidden-quadratic reversal shows that nonlinear representation mismatch remains consequential.
W3. Broader convergence conditions
Theorem 6 replaces uniform features with bounded candidate distributions whose pairs continue informing every parameter direction. It raises the sufficient true-logit bound from 1.763 to 3.049. The proof controls estimation-error growth, then establishes enough information growth for the unchanged recursive mean to approach the true preference weights and acceptance threshold. Independent arrivals, correctly specified stationary logistic feedback and continuing quotas remain explicit conditions.
Appendix H gives the theorem and proof, and Appendix H.3 covers suitable absolute-only policies and checks the native-feature scope. Consistency validates the estimator under these conditions. The finite-budget experiments identify when mixed feedback improves learning.