acceptodds
Under review as a conference paper at ICLR 2027

PersonaBias: Personalized Chain-of-Thought Optimization for Mitigating Personalized Bias in Large Language Models

Abstract

Conservative debiasing can reduce harmful outputs but may also lead to refusals or safe, generic responses that do not adequately address legitimate questions. User context can help assess the relevance and appropriateness of explanations while maintaining shared standards of factual accuracy and non-discrimination. We propose PersonaBias, a framework for personalized debiasing through user-conditioned reasoning selection before final answer generation. Strategy-guided generation produces candidate chains of thought with different explanatory perspectives. A User-Conditioned Reasoning Evaluator uses bidirectional cross-attention to model interactions between user profiles and candidate trajectories. It distinguishes three types of reasoning: biased, acceptable but insufficiently informative, and unbiased and informative. Candidates are ranked by their expected quality scores, and the highest-scoring trajectory guides the base LLM in generating its final response. Only the evaluator is trained; the base LLM remains frozen. Experiments on FairMT and BBQ show that PersonaBias improves the balance between bias mitigation and response utility. We further validate its effectiveness on a real-world, human-annotated dataset.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Personalized Debiasing, Chain-of-Thought Reasoning, Social Bias, Personalized Alignment, Inference-time Reranking

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AgRKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PersonaBias, a personalized debiasing framework. It generates strategy-guided chain-of-thought candidates and selects the optimal trajectory via a User-Conditioned Reasoning Evaluator, with the base LLM kept frozen. Its key contribution is formulating personalized debiasing as user-conditioned reasoning selection. The method is validated on FairMT, BBQ benchmarks and a human-annotated dataset to test bias-utility balance.

Critical strengths
  1. It designs a three-level reasoning quality metric to differentiate biased, under-informative and unbiased informative reasoning, which supports fine-grained evaluation beyond traditional binary bias detection.
  2. The frozen-base scheme applies to both open-weight and black-box LLMs, with deployment flexibility.
Critical weaknesses
  1. The work lacks sufficient justification for the reasoning selection module, with no thorough comparison to simpler alternatives such as personalized prompting.
  2. Most evaluations rely on LLM judges and synthetic labels, with limited real-user validation.
Questions for authors
  1. The abstract mentions "Only the evaluator is trained; the base LLM remains frozen". What is the core technical advantage of this frozen-base design compared with other frozen-base debiasing schemes in the same field?
  2. This work formulates personalized debiasing as a contextual reasoning selection task. Why is this paradigm necessary?
  3. Section 4.2 mentions "The source query x is available during candidate generation and label construction but is not a separate input to UCRE". Will this design cause the evaluator to fail to identify information omission of the reasoning relative to the original question?
  4. In the ablation study, the reranking ablation only uses random sampling as the control group. Why is simple heuristic ranking not set as a control?
  5. In Section 5.2, the comparison with BiasDPO only presents pairwise win-loss rates. Can the specific numerical differences between the two methods in the two dimensions of bias score and utility score be supplemented?
  6. The authors choose to freeze the two encoders when fine-tuning UCRE. What is the core consideration of this design? Can unfreezing the encoders further improve the F1 metric of preference matching?
  7. In the reference section, a considerable portion of the literature are only preprints. It is suggested to select the latest published literature from authoritative journals in the recent three years as much as possible.
4 comments

We sincerely thank the reviewer for the careful review, positive assessment of our work, and highly insightful suggestions. To address the concerns about reasoning selection, comparisons with simpler alternatives, and human validation, we have added comparative experiments and supplementary analyses, and clarified the framework’s motivation, evaluator inputs, and training settings. All related changes are highlighted in teal in the revised manuscript.

Response to W1: Comparison with personalized prompting

We added a simple personalized-prompting baseline that directly supplies the user profile and question to the same DeepSeek API model, without candidate generation or reranking. Across the three evaluated datasets, PersonaBias achieves win/tie/loss rates of 35.85%/36.66%/27.49%, 27.76%/42.05%/30.19%, and 30.19%/38.54%/31.27%, respectively. Thus, PersonaBias improves upon or matches personalized prompting in 72.51%, 69.81%, and 68.73% of comparisons—approximately seven out of ten cases across all three datasets. These comparisons complement our existing reranking ablation and support the practical value of reasoning generation and selection alongside user-profile conditioning.

Changes: We have added the personalized-prompting baseline to the comparison table in Section 5.2.

Response to W2: Human validation of labels and evaluations

Your emphasis on human validation is particularly valuable for establishing the credibility of our supervision and evaluation.

First, we conduct three rounds of blinded human review, each covering 120 training examples and 40 response pairs. Mean exact agreement is 82.22% with the training labels and 85.00% with automated win–tie–loss judgments. All training-label disagreements concern levels 1 and 2, with none involving the bias-risk category (level 0). Pairwise disagreements predominantly involve automated wins/losses judged as ties by humans. Figure 12 summarizes these findings.

Second, we evaluate an adapted UCRE on PrefLearn, using real participants’ existing preference annotations. On 96 user–question pairs from 24 held-out questions, PersonaBias achieves a macro-F1 of 0.45, outperforming random selection (0.37) and direct generation with profiles (0.34). These additions provide human-grounded evidence for label quality, automated evaluation, and preference matching.

Changes: We have added two complementary analyses in Appendix A.9 and Appendix A.13.

Response to Q1: Advantages of the frozen-base design

The frozen-base design makes PersonaBias applicable to both open-weight and API-based LLMs. Training is confined to the evaluator, reducing computational costs and avoiding the need for large-scale, high-quality supervision to retrain the base model. Keeping the base model unchanged also avoids parameter-update-induced capability degradation from extensive safety fine-tuning, supporting our objective of balancing task utility and bias mitigation.

Within this frozen-base setting, our core technical advantage lies in combining strategy-guided candidate generation with user-conditioned reasoning selection. Multiple strategies broaden the explanatory perspectives available in the candidate pool. UCRE then jointly assesses bias and explanatory sufficiency, distinguishing safe but generic reasoning from unbiased, informative reasoning. Selecting a trajectory before final answer generation allows the framework to retain useful explanations while controlling bias, extending the role of personalization beyond direct prompting.

Changes: We have clarified the deployment advantages of the frozen-base design and the technical contribution of combining strategy-guided generation with user-conditioned reasoning selection in the opening paragraph of Section 4.

Response to Q2: Motivation for contextual reasoning selection

Our motivation is to balance personalized bias mitigation and response utility. Conservative debiasing can suppress useful explanations, while an informative response must also account for its relevance and appropriateness to the current user.

The generate-then-select paradigm makes this trade-off explicit. Strategy-guided generation first produces alternative explanatory trajectories, preserving different ways to address the question. The evaluator then assesses their bias and explanatory sufficiency in the user’s context, selecting an unbiased, informative trajectory to guide the final answer. This separation allows the framework to explore useful explanations before deciding which best balances bias mitigation and utility for the current interaction.

Changes: We have added a brief explanation at the beginning of Section 3.2, clarifying why contextual reasoning selection makes the bias–utility trade-off explicit before final answer generation.

Response to Q3: Query omission and sufficiency assessment

In the original design, UCRE assesses explanatory sufficiency using task context conveyed through candidate trajectories. The query conditions candidate generation, and the prompt asks each trajectory to restate the task and essential context. On held-out FairMT, UCRE-selected trajectories covered 74.11% of the final query’s IDF-weighted content words on average.

To directly examine the effect of access to the complete question, we trained a query-aware variant that prepends the query to each trajectory, with matched training conditions, identical candidate pools, and the same selection rule. Explicit query access changed US from 1.757 to 1.744, BS from 1.844 to 1.825, and task completion from 80.68% to 80.77%.

These small average changes are consistent with the substantial task context already conveyed through the trajectories and support the effectiveness of the original input design in the evaluated setting.

Changes: We have added the coverage analysis and controlled query-access comparison to Appendix A.10.

Response to Q4: Comparison with heuristic ranking

We added a heuristic Similarity baseline that ranks candidate CoTs by their embedding cosine similarity with the user profile, using the same candidate pool.

MethodBS ↑US ↑Task completion ↑
Random1.7341.57775.58%
Similarity1.7251.72874.12%
UCRE1.8441.75780.68%

UCRE achieves the best performance across all three metrics. Compared with Similarity, it improves BS by 0.119, US by 0.029, and task completion by 6.56 percentage points, demonstrating the advantage of our learned reasoning selection.

Changes: We have added the heuristic selection comparison to Section 5.7.

Response to Q5: Numerical bias–utility comparison with BiasDPO

The bias–utility comparison is presented in Appendix A.8, Figure 11. For clarity, we supplement the exact numerical results below. Both metrics range from 0 to 2, with higher values indicating better performance.

BackboneMethodBS ↑US ↑
Llama3-8BBiasDPO1.551.29
Llama3-8BPersonaBias1.631.35
Qwen-3BBiasDPO1.451.10
Qwen-3BPersonaBias1.481.32

PersonaBias improves both dimensions over BiasDPO: BS and US increase by 0.08 and 0.06 on Llama3-8B, and by 0.03 and 0.22 on Qwen-3B, respectively. These results demonstrate stronger bias mitigation together with higher response utility across both backbones.

Response to Q6: Encoder freezing and preference adaptation

In the main experiments, we fine-tune the BGE user encoder end to end, keep the DeBERTa-v3-large CoT encoder frozen, and jointly train the projection, attention, and classification layers. Freezing both encoders applies only to the PrefLearn adaptation experiment in Appendix A.9. The base LLM used for generation remains frozen in both settings.

PrefLearn adaptation uses only 336 training user–question pairs from 84 questions. We freeze both encoders to reuse the representations learned during original UCRE training and limit overfitting on this small adaptation set. The projection, attention, and reinitialized classification layers are trained to adapt the evaluator to human-annotated preferences. User profiles continue to condition evaluation through their representations and cross-attention interactions.

Changes: We have explicitly added the main training configuration to Section 5.1 and clarify the adaptation-specific setting and rationale in Appendix A.9.

Response to Q7: Publication status of references

We have updated preprint references to their peer-reviewed publication versions. The revised bibliography contains only eight remaining arXiv/CoRR entries. Four of the eight remaining arXiv/CoRR entries are foundation-model technical reports.

Thank you for your rebuttal, I will keep current positive score.

Thank you for your detailed insights, which have significantly helped improve the quality of our paper.

Reviewer_cKUAOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Debiasing methods can hurt LLMs performance, as they are more prone to refusals even to harmless questions. To mitigate this issue, this paper proposes to condition on extra user information to avoid unnecessary safty regulations. They claim that such proposed method, Persona Bias, balances bias mitigation and answer quality better on the FairMT and BBQ benchmarks.

Critical strengths
  • It is an interesting claim that LLMs should be adaptive to different contexts when deciding if an answer is harmful or biased. It makes sense that we should not overly trade performance for safety.
  • In Figure 3, they demonstrate that their methods achieve the Pareto frontier of bias and utility.
Critical weaknesses
  • If the answer is dependent on the user's information, will this introduce a new fairness issue, where LLM answer quality differs across users?
  • The alignment tax may be distributed unevenly across different users: some groups will receive less harmful and less useful responses. Yet, bias and utility are reported on average over user profiles. It is suspected that such numbers will depend on the user distribution of the test set. How is the utility vs. bias performance per group?
  • the risk of jailbreaking: can a user determine u simply by claiming membership in a group during the conversation? If so, then an attacker can take advantage of the evaluator that is more permissive for certain profiles.
  • It is unclear how the ground truth labels are constructed for training the user-conditioned reasoning evaluator.
Questions for authors
  • How do the CoTs differ when sampled with versus without user information? This comparison would help clarify whether the additional user information actually affects answer generation.
  • How are ground-truth labels constructed for training the user-conditioned reasoning evaluator?
7 comments

We sincerely thank the reviewer for the detailed and thoughtful feedback on fairness across user profiles, identity-based jailbreaking, and training-label construction. To address these concerns, we have added a group-wise analysis and a controlled comparison of generation with and without user profiles, clarified our shared quality criteria and label construction, and discussed safeguards against identity manipulation. All related changes are highlighted in red in the revised manuscript.

Response to W1: Response-quality fairness across users

Thank you for highlighting the fairness implications of user-conditioned responses. PersonaBias trains a shared evaluator using the same three-level quality criteria across all user profiles. Factual accuracy, non-discrimination, and task usefulness remain common requirements. User context guides explanatory relevance and emphasis, helping preserve useful, supported information that overly conservative debiasing may otherwise omit.

To examine performance across users, we added a group-wise analysis covering age, gender, race/ethnicity, and religion. Group-level mean US ranges from 1.71 to 1.85, and BS from 1.66 to 1.86, with higher scores indicating better performance. These results support PersonaBias’s ability to preserve response utility while mitigating bias across the evaluated demographic groups.

Changes: We have clarified the shared quality criteria in Section 4.2 and added the group-wise analysis in Appendix A.11.

Response to W2: Group-wise bias–utility balance

We added a group-wise analysis on 371 held-out FairMT questions, grouping the assigned user profiles by age, gender, race/ethnicity, and religion.

Profile dimensionGroup US range ↑Group BS range ↑Equal-group-weighted US ↑
Age1.76–1.831.67–1.841.79
Gender1.73–1.811.72–1.861.79
Race/ethnicity1.71–1.841.75–1.851.79
Religion1.74–1.851.66–1.851.80

PersonaBias maintains high utility and bias scores across all evaluated groups. Giving each group equal weight yields nearly identical utility scores across the four dimensions (1.79–1.80), showing that the observed utility is maintained when demographic groups contribute equally. Thus, the favorable bias–utility balance extends across groups and is not confined to the aggregate results.

Changes: We have added this group-wise analysis and the corresponding results to Appendix A.11 of the revised manuscript.

Response to W3: Identity-based jailbreaking risks

A claimed group identity may influence the inferred context (u), making identity-based jailbreaking a relevant security consideration. Defending against such manipulation is complementary to our objective: PersonaBias uses user context to preserve useful, appropriate explanations that overly conservative debiasing may otherwise suppress. Group membership is not intended to grant permission for harmful content.

Identity manipulation can be addressed through additional safeguards, such as profile validation, prompt-injection defenses, and profile-independent safety checks. Since PersonaBias keeps the base LLM frozen and operates through candidate generation and selection, these safeguards can be integrated into the pipeline, allowing personalized reasoning selection to work alongside dedicated jailbreak defenses.

Changes: We have added this security consideration to the Conclusion, highlighting profile-independent safeguards against identity-based jailbreaking as a direction for future work.

Response to W4 & Q2: Training-label construction

We have expanded Appendix A.1 to clarify the training data and label construction, including a condensed generation prompt. Using FairMT and CREHate, we generate three contrasting reasoning trajectories for each question–profile pair: biased reasoning (0), unbiased but insufficient reasoning (1), and unbiased, informative reasoning (2). The trajectories undergo automated screening and manual review for consistency with their target labels. Retained examples inherit these labels, producing 6,000 training instances through target-guided generation and subsequent quality control.

Changes: We have expanded Appendix A.1 to clarify the training data and label construction.

Response to Q1: Effects of user profiles on generated reasoning

We conducted a controlled comparison on 100 sampled questions, generating responses with and without user profiles. Providing profiles increased final-answer utility from 1.72 to 1.84, while task completion remained similar (85.71% vs. 84.74%). These results indicate that user information primarily improves the usefulness of the generated explanations, with little change in task completion.

Changes: We have added this comparison to Appendix A.12 of the revised manuscript.

Thanks for your response. I still have one question:

Providing profiles increased final-answer utility from 1.72 to 1.84, while task completion remained similar (85.71% vs. 84.74%)

Is the US score judged by LLMs? If so, with 100 sampled questions, it is difficult to judge whether this difference is statistically meaningful. Can you also report the variance?

Reviewer_jcSaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PersonaBias aims to mitigate bias while preserving response utility through user-conditioned reasoning selection. A frozen model generates strategy-diverse reasoning candidates. UCRE evaluates each candidate against user context using three classes: biased, acceptable but insufficiently informative, and unbiased and informative. The highest expected-score candidate guides final generation. Experiments cover FairMT and BBQ. An appendix separately adapts the evaluator to human preference data.

Critical strengths
  1. The problem formulation is practically meaningful. The paper addresses information loss from conservative debiasing rather than merely checking discriminatory expressions. Three-level supervision distinguishes generic caution from sufficient explanation while allowing warranted uncertainty.

  2. Some controls are informative. Single matches the candidate budget, while Random shares the candidate pool. These comparisons help separate strategy generation from selection. Keeping the base model frozen also supports deployment through black-box APIs.

Critical weaknesses
  1. The independent contribution of personalization remains unclear.

    Random tests learned selection against random choice. It does not establish that profiles are necessary. Removing cross-attention retains user information and changes model capacity. A profile-agnostic selector with matched candidates, supervision, and training is missing. The gains may therefore mainly reflect generic quality selection.

    Appendix A.9 retrains the prediction head using F1 and ranking supervision. It supports adapted preference matching, rather than directly validating the original debiasing evaluator. Profile-specific gains must be separated from generic reranking gains.

  2. Assessment of task sufficiency and bias risk remains insufficiently validated.

    UCRE judges explanatory sufficiency without receiving the query. For example, “five videos cannot establish a group-level tendency” addresses evidential sufficiency but not what the user should tell their group chat. The evaluator may miss requirements omitted from the candidate.

    The score R = p₁ + 2p₂ also permits quality gains to offset predicted bias risk. For example, (0, 1, 0) scores 1, while (0.4, 0, 0.6) scores 1.2. This is not an observed bias rate, but it exposes a difference from the judge's bias-priority rule. Same-pool comparisons with query-aware evaluation and bias-priority selection should report final-answer bias and utility.

Questions for authors
  1. With matched candidate pools, supervision, and training conditions, how much does UCRE improve over a profile-agnostic selector? Are gains concentrated on context-relevant questions?

  2. On the same candidate pools, how do explicitly providing the query to UCRE and adopting bias-priority selection each affect final-answer bias and utility?

8 comments

We sincerely thank the reviewer for the detailed and thoughtful feedback on the contribution of user profiles and the evaluator’s input and scoring design. To address these concerns, we have added controlled comparisons with profile-agnostic and query-aware evaluators, examined bias-priority selection, and clarified our motivation and scoring formulation. All related changes are highlighted in blue in the revised manuscript.

Response to W1 & Q1: Independent contribution of user profiles

We have clarified the motivation in the Introduction and added a matched profile-agnostic comparison.

1. Clarified role of user profiles. We have revised the Introduction to explain that user profiles provide context for assessing an explanation’s relevance and sufficiency. Our goal is to retain useful, supported analysis that conservative debiasing may otherwise omit, while maintaining shared standards of factual accuracy and non-discrimination.

2. Matched comparison. We added a profile-agnostic selector (PA) with the same architecture, supervision, training budget, and random seeds as UCRE. PA receives a constant profile during training and inference, whereas UCRE receives the actual profile. All selectors operate on identical profile-conditioned candidate pools. This comparison isolates the additional contribution of profile access at selection time. On 371 FairMT test questions, PA improves task completion over uniform random selection by 4.83 percentage points, and UCRE provides a further 0.27 percentage points. We also evaluate final-answer utility and bias avoidance under the corresponding user context. Both scores range from 0 to 2, with higher values indicating better performance.

SelectorTask completion ↑US ↑BS ↑
Random selection (RND)75.58%1.5771.734
Profile-agnostic selector (PA)80.41%1.6931.725
UCRE80.68%1.7571.844

Relative to PA, UCRE improves US by 0.064 and BS by 0.119. These results distinguish the completion benefit of learned reranking from the additional bias and utility gains associated with profile-conditioned selection. They support our objective of preserving useful assistance for the current user while improving bias avoidance.

3. Role of the PrefLearn experiment. Appendix A.9 provides complementary evidence for adapted preference matching: the evaluator is adapted with a reinitialized prediction head using F1 and ranking supervision. The new FairMT comparison retains the original debiasing supervision and directly evaluates the contribution of profile-conditioned selection.

Changes: We have clarified the motivation in the Introduction, added the matched selector comparison in Section 5.7, and clarified the adaptation protocol and purpose of PrefLearn in Appendix A.9.

Response to W2 & Q2: Query access and bias-priority selection

We address these points through controlled same-pool comparisons and clarifications of the scoring formulation.

1. Explicit query access. In the original design, the query conditions candidate generation, and the prompt asks each trajectory to restate the task and essential context. Across the FairMT test questions, UCRE-selected trajectories cover 74.11% of the final query’s IDF-weighted content words on average. This provides a lexical measure of the query context retained in the trajectories.

To examine the effect of complete query access, we trained a query-aware evaluator that prepends the query to each trajectory under matched training conditions. Holding candidate pools and expected-score selection fixed, explicit query access changes US from 1.757 to 1.744, BS from 1.844 to 1.825, and task completion from 80.68% to 80.77%. Thus, query access produces a small completion increase alongside slightly lower average utility and bias-avoidance scores in this comparison.

2. Scoring semantics and bias-priority selection. The reviewer’s example correctly identifies a distinction between maximizing expected quality and prioritizing predicted bias risk. Class 2 requires both unbiasedness and explanatory sufficiency; detailed but biased reasoning belongs to class 0. The expected score (R=p1+2p2R=p_1+2p_2) aggregates predictions over these ordered classes, without imposing a separate constraint on (p0p_0).

We therefore compared expected-score selection with strict bias-priority selection, which minimizes (p0p_0) and uses (R) only to break ties. Across 371 test questions, three training seeds, and five candidates per pool, 32 of 11,130 unordered within-pool pairs (0.29%) exhibit both higher (p0p_0) and higher (R) for the same candidate under the original evaluator. Switching to strict bias priority changes 14 of 1,113 selections (1.26%) with the original evaluator and one selection (0.09%) with the query-aware evaluator.

UCRE inputSelection ruleUS ↑BS ↑Task completion ↑Changed selections
Original inputExpected score (R)1.7571.84480.68%—
With explicit queryExpected score (R)1.7441.82580.77%—
Original inputStrict bias priority1.7571.84480.77%14/1,113
With explicit queryStrict bias priority1.7441.82580.77%1/1,113

Strict bias priority leaves final-answer US and BS unchanged at the reported precision under both input settings. Task completion increases by 0.09 percentage points with the original evaluator and remains unchanged with the query-aware evaluator. These comparisons quantify the small aggregate effect of changing the selection rule in the evaluated candidate pools.

Changes: We have added Appendix A.10, detailing the experimental objectives, matched comparison settings, coverage analysis, and final-answer results for query access and bias-priority selection. All corresponding additions are highlighted in blue.

Thanks for the rebuttal and the new experiments. The rebuttal addresses some of my questions about evaluation. My main concern is still why demographic profiles are needed.

So, I want to ask additional questions:

Could conditioning on user profiles make the model more sycophantic?

It seems like common sense that debiasing should take priority over helpfulness. Is personalized debiasing really the right direction for the community?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.