acceptodds
Under review as a conference paper at ICLR 2027

Learn the User, Serve Them Better: From User Models to Personalized LLM Decisions

Abstract

An online personalized LLM learns from user feedback. While pairwise questions reveal which response a user prefers, absolute acceptability questions ask whether a response is good enough. With a limited feedback budget, our proposed DUAL (Dual-feedback User-Adaptive Learning) learns each user's preferences and acceptance threshold to decide whether to serve either response or abstain. An established information-gain score selects the question type, while a calibrated cutoff and quota completion determine when to ask. Under independent uniform candidate features, a correctly specified logistic user with moderate true logits, and continuing answer quotas, we prove that DUAL's original recursive estimates converge almost surely to the true preference weights and acceptance threshold. We also characterize information coverage and bound the online estimate's distance to a joint fit. A controlled development simulation isolates question choice: at identical question times and 940 answers, information selection achieves 15.85% area under the serving-mistake learning curve (AUC), versus 16.45% for a separately calibrated fixed allocation. A common-calibration development study compares complete policies at 940 answers: DUAL achieves 14.735% AUC versus 15.206% for adaptive acceptability-only questioning. These simulations use PersonalLLM features and controlled acceptability feedback. Exploratory PRISM results leave the benefit of mixing feedback after population initialization unresolved. Community Alignment supports personal choice-probability prediction and information-based question selection over entropy. The complete policy's human serving benefit remains open.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
LLM personalization, preference learning, active learning, Bayesian experimental design, online learning, selective prediction, human feedback

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Kq5gOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Standard LLM alignment learns relative preferences between candidate responses, which can fail when both candidates are not acceptable. This paper proposes a method called DUAL aiming to address this by maintaining a joint online Gaussian belief over a user's feature preferences and a latent acceptance threshold to support a three-way decision space, consisting of Serve A, Serve B, or Abstain. Under a constrained query budget, their method uses local Bayesian D-optimality to pick between pairwise and absolute acceptability questions.

Critical strengths
  • The formulation of acceptance of abstention in LLM serving is interesting and to my knowledge, novel. The idea that preferring a response over another doesn’t mean the preferred response is acceptable is intuitive and correct and is indeed an issue with the standard pairwise preference alignment. The proposed framework Is also intuitive and interesting. It combines an acceptance threshold, abstention, and two feedback channels in one framework. The theoretical analysis provided for the unmodified recursion is also solid. In particular, the almost-true convergence for the original online recursion under adaptive query selection and continuing answer quotas is sound and strong.

  • The proposed framework is lightweight and efficient. The DUAL method uses one shared belief for serving, question selection, and timing. The information score also has a closed form from the determinant lemma, so the cost per step is low.

  • The experimental report benefits from an exceptional rigour and transparency which I appreciate. The authors clearly distinguish development exploration from confirmatory evaluations, apply strict Bonferroni adjustments to multi-contrast hypothesis families, and most importantly, openly report unresolved and null results instead of over claiming about the empirical wins. This rigorous analysis enhances the integrity and trustworthiness of the work and its findings. The controlled ablations that isolate the contributions (Table 1 and 2) are particularly beneficial.

Critical weaknesses
  • While I appreciate the transparency and integrity in the reported results, my understanding from the reported results is that when tested on actual human ratings in PRISM with a population-initialized prior, the serving AUC difference between mixed feedback which is 58.14% and absolute-only feedback 57.86% with an overlapping CI is statistically unresolved. It seems that the majority of the performance comes from the population prior and not the personal feedback, so the empirical justification of dual-channel elicitation is unclear to me.

  • The simulation does not appear to be conducted in the most accurate way. The users are generated from the same linear-logistic model that the learner assumed. Acceptance threshold are quantities of the same utility, and answers are drawn from Eq 3 and Eq 4 at unit temperature. The learner is thus correctly specified by design. The misspecification test also confirms the fragility. A hidden quadratic preference component reverses the timing advantage, and recalibration doesn’t salvage it entirely. In practical applications, real users have non-linear, context-dependent, with different levels of noise, and inconsistent preferences. Applying the same fixed noise scale is also an inappropriate decision that favours the information score’s assumptions.

  • The theorem 3 has strict assumptions that don’t seem realistic to me. The candidate features are uniform over [-1,1]^d, a correct specified linear logistic user model is assumed, and true logits bounded by 1.76.In reality, response embeddings are rarely uniform, and real-world preferences may have non-linearities and extreme logits when there are near-deterministic choices.

Questions for authors

Please find my comments in the weaknesses part. In addition to those, Have you conducted any analysis on the sensitivity of the AUC gains to the assumed noise temperature, which is fixed at 1 for both channels?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Kq5g

Thank you for recognizing the formulation, lightweight computation and convergence analysis. Your concerns motivated clearer population controls, new feedback-noise tests and broader sufficient conditions. These additions are marked in blue.

W1. Population initialization and the evidence for mixing

The population prior is a strong contributor on PRISM. On the same 90-participant cohort, population-only, mixed and acceptability-only models achieve 58.89%, 58.14% and 57.86% serving-mistake AUC. Mixed minus population-only is -0.75 points [-3.32, +1.81], and mixed minus acceptability-only is 0.28 [-0.50, +1.06], using exploratory marginal 95% intervals. Appendix G.6 places these controls beside the updating methods and reports trivial-policy comparisons.

We frame PRISM as a component evaluation using recorded ratings and a fixed acceptability threshold. The stronger evidence for adaptive mixing comes from the prospectively frozen simulation: DUAL achieves 14.71% versus 15.94% for adaptive acceptability-only on 200 fresh profiles, with adjusted difference -1.229 points [-1.866, -0.592]. Section 6.4, Table 3 and Appendix G.2 report that comparison.

W2 and noise question. Unequal and changing feedback reliability

Feedback reliability changes which channel is worth buying. We varied true acceptability/pairwise temperatures across (1,1)(1,1), (1,2)(1,2) and (2,1)(2,1) while keeping the learner at (1,1)(1,1). Temperature 2 makes answers less decisive. Each policy was separately calibrated, then assessed on the same 120 fresh profiles with a 940-unit cap over 9,402 interactions.

At equal prices, noisier acceptability feedback gives a DUAL-minus-acceptability-only AUC difference of -4.465 points [-5.704, -3.274]. Noisier pairwise feedback instead favors acceptability-only by 1.326 points [+0.292, +2.360]. With noisier acceptability feedback, the difference remains -3.168 points at twice the pairwise price. Table 50 reports the full 15-comparison adjusted family for the original score with cost-cap timing.

Appendix G.5 also tests unannounced noise changes without recalibration. The learner has no reliability parameter. These tests address feedback mismatch, while the retained hidden-quadratic reversal shows that nonlinear representation mismatch remains consequential.

W3. Broader convergence conditions

Theorem 6 replaces uniform features with bounded candidate distributions whose pairs continue informing every parameter direction. It raises the sufficient true-logit bound from 1.763 to 3.049. The proof controls estimation-error growth, then establishes enough information growth for the unchanged recursive mean to approach the true preference weights and acceptance threshold. Independent arrivals, correctly specified stationary logistic feedback and continuing quotas remain explicit conditions.

Appendix H gives the theorem and proof, and Appendix H.3 covers suitable absolute-only policies and checks the native-feature scope. Consistency validates the estimator under these conditions. The finite-budget experiments identify when mixed feedback improves learning.

I appreciate the authors' rebuttal and maintain my positive assessment of the work. I thank the authors for having been transparent and thorough in adding ablations and caveats, rather than overclaiming.

AuthorsOpenReview ↗

Thank you for your feedback

Dear Reviewer Kq5g,

Thank you for taking the time to read our response and for maintaining your positive assessment. We’re glad the added analyses and clarifications were helpful. Your feedback helped us improve the paper and make both the results and their limitations clearer.

Reviewer_zMTmOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the problem of adaptively decide when to request user feedback and whether to ask an absolute acceptability question or a pairwise preference question. The authors proposed a conditionally theoretically guaranteed framework, DUAL, to jointly learn the user's preference model and acceptance model, which outperforms in the main simulation experiment comparing to four baselines. The paper also provide detailed study of the contribution of two components: adaptive timing and adaptive question-type selection.

Critical strengths
  • This article formulated an interesting guaranteed DUAL framework: under a pre-specified user model, the framework learns the user preference and acceptance threshold to decide whether to serve either response or abstain. Under explicit user-model and candidate-distribution assumptions, this framework is also theoretically guaranteed and the guarantee particularly addressed the problem of non-independent distribution induced by adaptive questioning.

  • DUAL achieves lower mistake AUC than the four reported baselines in the main simulation experiment in Section 6. The authors also provides detailed study of the contribution of adaptive timing and information-based question-type selection, respectively.

Critical weaknesses
  • W1: Section 6.2 compares DUAL against four baselines and additionally reports a comparison with adaptive absolute-only questioning. However, the corresponding adaptive pairwise-only baseline is missing, leaving the comparison against single-channel adaptive querying incomplete. The paper does not explicitly explain this omission.

  • W2: The advantage of information-based timing over confidence-based timing remains limited, even in this model-matched simulations. As reported in Section 6.3 and Table 22, confidence timing achieves lower mean bank error at the reported checkpoints through 100 answers, whereas DUAL performs better at 250, 500, and 940 answers. These checkpoints are prefixes of policies configured for 940 answers, rather than independently calibrated short-budget experiments. Thus, the results do not establish a clear overall advantage over the simpler confidence-based timing baseline.

  • W3: Related to W2, the relative performance of DUAL and its baselines across different feedback budgets remains insufficiently characterized. Although the paper reports budget sensitivity for DUAL, it lacks a systematic comparison with policies including baselines and adaptive absolute/pairwise-only questioning across budget levels with each policy calibrated for the corresponding budget. Such comparisons would clarify whether DUAL’s benefits is robust across budget levels.
Questions for authors
  • Please address the weaknesses.
4 comments
AuthorsOpenReview ↗

Response to Reviewer zMTm

Thank you for identifying the missing adaptive pairwise-only baseline and the need for budget-specific calibration. We completed both comparisons and added the results in blue in Section 6.4 and Appendix G.1.

W1. Adaptive pairwise-only comparison

DUAL reduces serving-mistake learning-curve AUC by 2.46, 4.16, 6.51 and 8.02 percentage points at budgets of 94, 188, 470 and 940 answers, respectively. All four paired intervals exclude zero after adjustment across the 16 primary comparisons. Acceptability feedback supplies the missing information: pairwise likelihoods cancel the acceptance threshold, whereas acceptability answers directly inform whether a response meets the user's standard. Table 45 reports the complete contrasts.

Full-horizon serving-mistake AUC (%, lower is better), rounded to two decimals. Each comparator below is adaptive:

AnswersDUALAcceptability-onlyPairwise-onlyConfidenceBALD
9428.8929.5131.3530.3728.78
18824.3925.0328.5525.3824.87
47018.7819.7625.2919.4318.77
94014.9215.4622.9315.4414.93

W2. Why use information timing when confidence timing is competitive?

DUAL values what an available question can teach about the user. Confidence timing measures uncertainty about the current serving action. Two nearly tied responses can leave that action uncertain even when parameter uncertainty is small and another answer adds little information. DUAL combines predicted answer variability with parameter uncertainty along the question's direction, linking both channel choice and timing to learning. Both policies serve before asking, so feedback benefits subsequent interactions.

At the two smallest separately calibrated budgets, DUAL achieves 28.89% versus 30.37% for confidence timing at 94 answers, and 24.39% versus 25.38% at 188 answers. DUAL has lower mean AUC at all four budgets, but every adjusted interval against confidence timing includes zero. At 940 answers, the difference is -0.520 points [-1.273, +0.218]. These results support competitive performance across budgets. They do not establish statistical superiority over confidence timing. Appendix G.1 explains the two triggers and reports the full comparisons.

W3. Separate calibration and computation across budgets

We compared eight policies at each budget, calibrating each adaptive policy separately on 100 simulated profiles and evaluating all policies on the same 200 fresh profiles over 9,402 interactions. Candidate streams, initialization, serving and updates were shared. Tables 44 to 46 include all adaptive and periodic controls. These are policies designed for each budget, rather than prefixes of a larger-budget run.

Every adaptive policy receives the same five-candidate cutoff search and calibration objective. Cutoffs are frozen before assessment profiles are generated, so assessment outcomes cannot influence tuning.

DUAL and BALD have closely matched observed AUCs, including 14.92% and 14.93% at 940 answers. DUAL's closed-form score avoids BALD's numerical integration. In a separate six-profile, three-repetition timing check, optimized 24-point BALD used a median 1.24 times DUAL's policy/simulator CPU time, excluding evaluation-bank scoring, response generation and participant latency. This supports a practical computation advantage over the tested BALD implementation alongside competitive observed accuracy. Appendix G.3 gives the timing protocol and learning-stage comparisons.

Together, the added adaptive pairwise-only baseline and separately calibrated four-budget study directly address W1 and W3. DUAL improves over adaptive pairwise-only at every budget, with all four adjusted intervals excluding zero.

AuthorsOpenReview ↗

Follow-up on the revised comparisons

Dear Reviewer zMTm,

Thank you again for your feedback. We wanted to follow up on the new comparisons in Section 6.4 and Appendix G.1. We’ve added the adaptive pairwise-only baseline and compared the policies at four budgets, with separate calibration for each adaptive policy. DUAL improves over adaptive pairwise-only at every budget after adjustment for multiple comparisons.

Could you let us know whether these additions address W1 and W3, and if any concerns remain about W2? We’d be happy to clarify anything further.

I sincerely thank the reviewer for their rebuttal, which has addressed my concerns. I would like to raise my score to weak acceptance.

AuthorsOpenReview ↗

Thank you for your feedback

Dear Reviewer zMTm,

Thanks for getting back to us! We’re glad the updates addressed your concerns. Your feedback really helped us improve the paper. Thanks again for your time.

Reviewer_L7RNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies online LLM personalization with abstention. Each interaction presents two responses with feature vectors x_A and x_B. The user is modeled by a linear utility vᵀx and an acceptance threshold b. The system serves A, serves B, or abstains, and the ideal action serves the better response only if it clears b.

Under a quota of 10 answers per 100 interactions, the system may ask one question: an absolute "good enough?" question, σ(vᵀx − b), or a pairwise "which is better?" question, σ(vᵀ(x_A − x_B)).

The proposed policy, DUAL, works in four steps:

It keeps a Gaussian belief over (v, b), updated by a one-step logistic Kalman/Laplace recursion. It serves the action with the highest posterior region probability. It scores the absolute question about the served response against the pairwise question with a local D-optimal gain, ½·log(1 + p(1−p)·zᵀΣz). It asks the higher-scoring question when that gain clears a calibrated cutoff, or when the quota forces a question. Theory gives almost-sure consistency of the recursion under uniform features, a correctly specified user, and a parameter bound of arcosh(3), plus conditional serving guarantees.

Simulations on PersonalLLM show DUAL lowering mistake AUC by 0.5–1.9 points at equal answer counts. It loses to absolute-only questioning when a pairwise question is charged for showing two responses, and on human data (PRISM) the benefit of mixing question types is unresolved.

Critical strengths

A well-posed, practical problem: acceptability and abstention on top of pairwise personalization. Careful, candid statistics. These include paired bootstrap intervals, multiplicity corrections, and matched answer budgets. Unfavorable results are reported openly (Tables 17, 18, 24). A convergence proof that looks correct (App. C.4). The paper is upfront that Prop. 2 specializes known results. Component evidence on human data. On Community Alignment, personal histories beat population-only and donor controls (Tables 31–32).

Critical weaknesses

Mixing helps only under equal answer counts. DUAL beats adaptive absolute-only by 0.47 points at equal answer counts. When a pairwise question is charged for showing two responses, it is worse by 1.33 points [+1.00, +1.65] (Table 18). On PRISM, whether mixing helps is unresolved. The simulator matches the learner's model. A hidden quadratic utility term reverses the timing advantage (Tables 17, 25). Unequal noise between the two channels isn't tested on PersonalLLM. The theory doesn't single out DUAL. It assumes uniform features and requires 2‖v*‖₁ ≤ arcosh(3) ≈ 1.76. Dirichlet-drawn users have ‖v‖₁ = 1, which gives 2 and violates that bound. Simpler policies appear amenable to the same proof. Narrow confirmatory evidence. The comparison against adaptive absolute-only was never confirmed on the held-out cohort. 98.6% of late questions are forced by the quota (Table 8). Safety signal. Compared with confidence-based timing, DUAL serves more often when neither response is acceptable (+0.345, Table 21). Missing baselines. There is no BALD/EPIG, Thompson-sampling, or LLM-native baseline.

Questions for authors

Does the proof of Thm. 3 adapt to adaptive absolute-only and periodic questioning? Do the Dirichlet users have ‖v‖₁ = 1 and so violate Eq. 36? What fraction of held-out users satisfy it? What happens under unequal true noise between the two question types, with the learner still using unit temperatures? Where between cost ratios of 1 and 2 does DUAL stop beating absolute-only? How does it compare with BALD-style mutual information and with Thompson sampling? Can you run a prospectively frozen comparison on fresh profiles? What mistake rates do trivial policies get on PRISM?

2 comments
AuthorsOpenReview ↗

Response to Reviewer L7RN

Thank you for the detailed questions. We added the comparisons and broader proof in blue. The grouped answers point to the revised manuscript for details. AUC denotes serving-mistake learning-curve area, with lower values better. Differences are percentage points.

Q1 to Q3. Broader convergence and Dirichlet users

Yes, suitable adaptive and periodic absolute-only policies also converge. Theorem 6 and Appendix H.3 extend the analysis to bounded candidate distributions with joint directional coverage. The proof controls estimation-error growth before establishing sufficient precision growth to force the recursive estimate toward the true user parameters. It retains independent arrivals, correctly specified stationary logistic feedback and continuing information. The finite-budget comparisons establish when mixing helps.

Normalized Dirichlet weights give 2∥v∥1=22\lVert v\rVert_1=2, exceeding the submitted bound 1.763. The wider bound 3.049 accommodates these weights with cube-bounded features and a utility-quantile threshold. Native standardized features need separate checks: all 100 inspected calibration profiles have some wider-bound violations. The original held-out satisfaction fraction remains unmeasured. Appendix H.3 states this scope explicitly.

Q4 and Q5. When mixed feedback earns its cost

Reliable preference feedback can justify its extra cost. We tested 120 fresh profiles with a 940-unit cap, retaining DUAL's original score with cost-cap timing and unit learner temperatures. When true acceptability/pairwise temperatures are (2,1)(2,1), DUAL minus acceptability-only is -4.465 points at equal prices and -3.168 points [-4.694, -1.679] at twice the pairwise price. Reversing the temperatures favors acceptability-only by 1.326 points at equal prices. Table 50 reports all 15 price/noise comparisons and adjusted intervals.

Under matched noise, the mean changes sign between tested prices 1.5 and 1.75, while all five adjusted intervals include zero. The grid shows cost sensitivity without establishing a critical price. The submitted exposure-cost loss is retained. Appendix G.5 defines the modeled costs and separate calibration.

Q6 and Q7. Stronger baselines and fresh-profile confirmation

After calibration on 30 profiles, we froze sources, protocols and cutoffs before generating 200 fresh simulated profiles. With 930 answers over 9,338 interactions, DUAL achieves 14.71% AUC versus 15.33% for serving-action EPIG and 15.94% for adaptive acceptability-only. Both differences exclude zero after adjustment across six comparisons. Confidence timing remains competitive. Table 3 and Appendix G.2 describe this known-prompt-distribution study, including EPIG's 64 reserved unlabeled reference prompts.

In the separate four-budget study, DUAL and BALD achieve 14.92% and 14.93% at 940 answers. Optimized BALD uses a median 1.24 times DUAL's policy/simulator CPU time in the sequential control, excluding bank scoring and response generation (Appendix G.3).

Thompson-style serving samples user parameters while retaining DUAL's feedback-scoring rule. In another 200-profile study with 940 answers, DUAL achieves 14.81% versus 19.88% AUC. The paired difference is -5.07 points, with 95% interval [-5.44, -4.69]. This supports integrated serving against this specified sampling adaptation (Appendix G.4, Table 49).

Q8 and the LLM baseline. PRISM controls

On the same 90-participant bank, always serving A/B gives 62.22%/61.67% mistakes, always abstaining 76.11%, uniform random A/B/abstain 66.67%, and uniform random A/B 61.94%. Population-only AUC is 58.89%, mixed feedback 58.14% and acceptability-only 57.86%. Population initialization supplies much of the observed benefit (Table 52).

With identical personal histories of zero to five answers, the explicit model and Qwen3-8B have mixed-feedback AUCs of 58.14% and 58.92%, with adjusted difference interval [-6.97, +5.33]. Absolute-only means favor Qwen. Appendix G.6 and Table 53 report both comparisons and the different representations and population initialization. These are recorded-feedback component evaluations.

Timing, serving errors and nonlinear mismatch

Quota completion sustains later feedback, consistent with the reported 98.6% late forcing rate, while channel choice can still adapt. Against confidence timing, DUAL reduces unnecessary abstentions by 0.459 points but increases serving when neither response is acceptable by 0.345 points. Tripling the unacceptable-service penalty reverses the mean ranking in a separate fixed-policy analysis (Appendix G.7). The adverse hidden-quadratic result is retained.

AuthorsOpenReview ↗

Follow-up on the additional results

Dear Reviewer L7RN,

Thank you again for the detailed questions. We wanted to follow up on our response and the additional results. These include the comparison on 200 fresh profiles with settings fixed before testing, the unequal-noise and feedback-cost experiments, and the broader convergence analysis (Section 6.4 and Appendices G.2, G.5 and H).

The fresh-profile results favour DUAL over both EPIG and adaptive acceptability-only after adjustment for multiple comparisons. We’ve also kept the unresolved PRISM results and the serving-error trade-off explicit.

Are there any concerns you feel we haven’t addressed yet? Please let us know if there’s a point you’d like us to explain in more detail.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.