- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents PersonaBias, a personalized debiasing framework. It generates strategy-guided chain-of-thought candidates and selects the optimal trajectory via a User-Conditioned Reasoning Evaluator, with the base LLM kept frozen. Its key contribution is formulating personalized debiasing as user-conditioned reasoning selection. The method is validated on FairMT, BBQ benchmarks and a human-annotated dataset to test bias-utility balance.
Critical strengths
- It designs a three-level reasoning quality metric to differentiate biased, under-informative and unbiased informative reasoning, which supports fine-grained evaluation beyond traditional binary bias detection.
- The frozen-base scheme applies to both open-weight and black-box LLMs, with deployment flexibility.
Critical weaknesses
- The work lacks sufficient justification for the reasoning selection module, with no thorough comparison to simpler alternatives such as personalized prompting.
- Most evaluations rely on LLM judges and synthetic labels, with limited real-user validation.
Questions for authors
- The abstract mentions "Only the evaluator is trained; the base LLM remains frozen". What is the core technical advantage of this frozen-base design compared with other frozen-base debiasing schemes in the same field?
- This work formulates personalized debiasing as a contextual reasoning selection task. Why is this paradigm necessary?
- Section 4.2 mentions "The source query x is available during candidate generation and label construction but is not a separate input to UCRE". Will this design cause the evaluator to fail to identify information omission of the reasoning relative to the original question?
- In the ablation study, the reranking ablation only uses random sampling as the control group. Why is simple heuristic ranking not set as a control?
- In Section 5.2, the comparison with BiasDPO only presents pairwise win-loss rates. Can the specific numerical differences between the two methods in the two dimensions of bias score and utility score be supplemented?
- The authors choose to freeze the two encoders when fine-tuning UCRE. What is the core consideration of this design? Can unfreezing the encoders further improve the F1 metric of preference matching?
- In the reference section, a considerable portion of the literature are only preprints. It is suggested to select the latest published literature from authoritative journals in the recent three years as much as possible.
4 comments
Response to Q3: Query omission and sufficiency assessment
In the original design, UCRE assesses explanatory sufficiency using task context conveyed through candidate trajectories. The query conditions candidate generation, and the prompt asks each trajectory to restate the task and essential context. On held-out FairMT, UCRE-selected trajectories covered 74.11% of the final query’s IDF-weighted content words on average.
To directly examine the effect of access to the complete question, we trained a query-aware variant that prepends the query to each trajectory, with matched training conditions, identical candidate pools, and the same selection rule. Explicit query access changed US from 1.757 to 1.744, BS from 1.844 to 1.825, and task completion from 80.68% to 80.77%.
These small average changes are consistent with the substantial task context already conveyed through the trajectories and support the effectiveness of the original input design in the evaluated setting.
Changes: We have added the coverage analysis and controlled query-access comparison to Appendix A.10.
Response to Q4: Comparison with heuristic ranking
We added a heuristic Similarity baseline that ranks candidate CoTs by their embedding cosine similarity with the user profile, using the same candidate pool.
| Method | BS ↑ | US ↑ | Task completion ↑ |
|---|---|---|---|
| Random | 1.734 | 1.577 | 75.58% |
| Similarity | 1.725 | 1.728 | 74.12% |
| UCRE | 1.844 | 1.757 | 80.68% |
UCRE achieves the best performance across all three metrics. Compared with Similarity, it improves BS by 0.119, US by 0.029, and task completion by 6.56 percentage points, demonstrating the advantage of our learned reasoning selection.
Changes: We have added the heuristic selection comparison to Section 5.7.
Response to Q5: Numerical bias–utility comparison with BiasDPO
The bias–utility comparison is presented in Appendix A.8, Figure 11. For clarity, we supplement the exact numerical results below. Both metrics range from 0 to 2, with higher values indicating better performance.
| Backbone | Method | BS ↑ | US ↑ |
|---|---|---|---|
| Llama3-8B | BiasDPO | 1.55 | 1.29 |
| Llama3-8B | PersonaBias | 1.63 | 1.35 |
| Qwen-3B | BiasDPO | 1.45 | 1.10 |
| Qwen-3B | PersonaBias | 1.48 | 1.32 |
PersonaBias improves both dimensions over BiasDPO: BS and US increase by 0.08 and 0.06 on Llama3-8B, and by 0.03 and 0.22 on Qwen-3B, respectively. These results demonstrate stronger bias mitigation together with higher response utility across both backbones.
Response to Q6: Encoder freezing and preference adaptation
In the main experiments, we fine-tune the BGE user encoder end to end, keep the DeBERTa-v3-large CoT encoder frozen, and jointly train the projection, attention, and classification layers. Freezing both encoders applies only to the PrefLearn adaptation experiment in Appendix A.9. The base LLM used for generation remains frozen in both settings.
PrefLearn adaptation uses only 336 training user–question pairs from 84 questions. We freeze both encoders to reuse the representations learned during original UCRE training and limit overfitting on this small adaptation set. The projection, attention, and reinitialized classification layers are trained to adapt the evaluator to human-annotated preferences. User profiles continue to condition evaluation through their representations and cross-attention interactions.
Changes: We have explicitly added the main training configuration to Section 5.1 and clarify the adaptation-specific setting and rationale in Appendix A.9.
Response to Q7: Publication status of references
We have updated preprint references to their peer-reviewed publication versions. The revised bibliography contains only eight remaining arXiv/CoRR entries. Four of the eight remaining arXiv/CoRR entries are foundation-model technical reports.
Thank you for your rebuttal, I will keep current positive score.
Thank you for your detailed insights, which have significantly helped improve the quality of our paper.
We sincerely thank the reviewer for the careful review, positive assessment of our work, and highly insightful suggestions. To address the concerns about reasoning selection, comparisons with simpler alternatives, and human validation, we have added comparative experiments and supplementary analyses, and clarified the framework’s motivation, evaluator inputs, and training settings. All related changes are highlighted in teal in the revised manuscript.
Response to W1: Comparison with personalized prompting
We added a simple personalized-prompting baseline that directly supplies the user profile and question to the same DeepSeek API model, without candidate generation or reranking. Across the three evaluated datasets, PersonaBias achieves win/tie/loss rates of 35.85%/36.66%/27.49%, 27.76%/42.05%/30.19%, and 30.19%/38.54%/31.27%, respectively. Thus, PersonaBias improves upon or matches personalized prompting in 72.51%, 69.81%, and 68.73% of comparisons—approximately seven out of ten cases across all three datasets. These comparisons complement our existing reranking ablation and support the practical value of reasoning generation and selection alongside user-profile conditioning.
Changes: We have added the personalized-prompting baseline to the comparison table in Section 5.2.
Response to W2: Human validation of labels and evaluations
Your emphasis on human validation is particularly valuable for establishing the credibility of our supervision and evaluation.
First, we conduct three rounds of blinded human review, each covering 120 training examples and 40 response pairs. Mean exact agreement is 82.22% with the training labels and 85.00% with automated win–tie–loss judgments. All training-label disagreements concern levels 1 and 2, with none involving the bias-risk category (level 0). Pairwise disagreements predominantly involve automated wins/losses judged as ties by humans. Figure 12 summarizes these findings.
Second, we evaluate an adapted UCRE on PrefLearn, using real participants’ existing preference annotations. On 96 user–question pairs from 24 held-out questions, PersonaBias achieves a macro-F1 of 0.45, outperforming random selection (0.37) and direct generation with profiles (0.34). These additions provide human-grounded evidence for label quality, automated evaluation, and preference matching.
Changes: We have added two complementary analyses in Appendix A.9 and Appendix A.13.
Response to Q1: Advantages of the frozen-base design
The frozen-base design makes PersonaBias applicable to both open-weight and API-based LLMs. Training is confined to the evaluator, reducing computational costs and avoiding the need for large-scale, high-quality supervision to retrain the base model. Keeping the base model unchanged also avoids parameter-update-induced capability degradation from extensive safety fine-tuning, supporting our objective of balancing task utility and bias mitigation.
Within this frozen-base setting, our core technical advantage lies in combining strategy-guided candidate generation with user-conditioned reasoning selection. Multiple strategies broaden the explanatory perspectives available in the candidate pool. UCRE then jointly assesses bias and explanatory sufficiency, distinguishing safe but generic reasoning from unbiased, informative reasoning. Selecting a trajectory before final answer generation allows the framework to retain useful explanations while controlling bias, extending the role of personalization beyond direct prompting.
Changes: We have clarified the deployment advantages of the frozen-base design and the technical contribution of combining strategy-guided generation with user-conditioned reasoning selection in the opening paragraph of Section 4.
Response to Q2: Motivation for contextual reasoning selection
Our motivation is to balance personalized bias mitigation and response utility. Conservative debiasing can suppress useful explanations, while an informative response must also account for its relevance and appropriateness to the current user.
The generate-then-select paradigm makes this trade-off explicit. Strategy-guided generation first produces alternative explanatory trajectories, preserving different ways to address the question. The evaluator then assesses their bias and explanatory sufficiency in the user’s context, selecting an unbiased, informative trajectory to guide the final answer. This separation allows the framework to explore useful explanations before deciding which best balances bias mitigation and utility for the current interaction.
Changes: We have added a brief explanation at the beginning of Section 3.2, clarifying why contextual reasoning selection makes the bias–utility trade-off explicit before final answer generation.