acceptodds
Under review as a conference paper at ICLR 2027

Revealing the Biases of Large Language Models by Inferring Prior Distributions

Abstract

Large Language Models (LLMs) have been known to exhibit a wide range of social biases including name-nationality associations, occupation-gender associations and racial stereotypes. Existing approaches to evaluating these biases typically involve measuring surface level behavior where a model is fed a dataset of prompts and its outputs are scored relative to ground truth answers. Inspired by prior elicitation in cognitive science, we propose a different perspective that reconstructs latent bias as a prior underlying the model's behavior. We formulate the problem as a Bayesian inverse problem where given prompt response observations on a bias benchmark dataset we infer a low-dimensional latent distribution that summarizes the model's stereotyping tendencies. This paper develops the mathematical foundation, a practical reconstruction algorithm, and experiments using the Bias Benchmark for QA (BBQ) benchmark to infer prior distributions of LLMs over responses in ambiguous social contexts.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Social Bias, Inverse Problems, Prior Elicitation, Bayesian Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bn99OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method for reconstructing latent bias of LLMs from their outputs. Different from traditional, singular bias scores, this approach characterizes how models systematically allocate probability mass across answer categories. Its main contribution is a Bayesian inverse-problem framework that decomposes observed candidate-answer log-probabilities into a shared latent preference bias, prompt-specific evidence shifts, and noises. The method uses MAP estimation and Laplace approximation to reconstruct a latent prior distribution. The paper validates the proposed method through three experiments: (1) Synthetic-data validation, which tests whether the algorithm can recover a known ground-truth distribution; (2) Caesar cipher experiments, which examine whether LLMs exhibit preferences over candidate shifts when informative input evidence is absent; and (3) BBQ experiments, which analyze social biases in ambiguous contexts and demonstrate how reconstructed distributions reveal behavioral differences that scalar bias scores may hard to imply.

Critical strengths
  1. The paper provides a mathematically principled framework with explicit theoretical formulation. The authors formulate latent preference inference as a Bayesian inverse problem, explicitly modeling the shared latent prior, prompt-dependent evidence, and observation noise. They further develop a MAP-based reconstruction algorithm and use Laplace approximation to quantify uncertainty. This is important because the method is grounded in a clear mathematical formulation rather than relying solely on empirical scoring.

  2. The paper offers a novel perspective on evaluating LLM biases beyond conventional benchmark scores. By reconstructing latent preference distributions, the method provides a more fine-grained characterization of how LLMs allocate probability mass across answer categories, revealing behavioral differences that may be obscured by scalar benchmark scores. The synthetic-data validation and experiments on Caesar ciphers and BBQ provide complementary evidence for the method's reconstruction capability and potential practical utility.

Critical weaknesses
  1. The generalizability and validity of the proposed reconstruction framework remain insufficiently established. The evaluation is primarily conducted on a single social-bias benchmark, BBQ, with the original answer space mapped to three categories. It remains unclear whether the mapping preserves the relevant bias structure, whether alternative mappings or datasets would yield consistent conclusions, and whether the framework remains effective with larger or unrestricted answer spaces

  2. The robustness and statistical reliability of the reconstructed preference distributions require more systematic evaluation. The paper does not systematically investigate how reconstructed distributions change with prompt templates, answer representations, sample sizes, or variations in observed output distributions. In particular, it remains unclear whether the inferred preferences are stable across semantically equivalent prompts and data subsets, and how reliably the method recovers latent preferences when the number and characteristics of observations change. If lack such robustness and sensitivity analyses, it is difficult to determine whether the observed differences reflect stable model behavior or artifacts of the experimental setup.

Questions for authors
  1. The BBQ experiments map the original answers to three categories: {S, A, U}. How sensitive are the reconstructed preference distributions to this mapping, and have the authors tested whether the method yields consistent conclusions on other bias benchmarks with the same {S, A, U} or a larger answer spaces?
  2. Have the authors evaluated how the reconstructed distributions change with the number of prompts, prompt templates, and answer representations? Specifically, do semantically equivalent prompts and different data subsets yield consistent preference estimates, and how does estimation uncertainty change as the sample size increases?
  3. The framework assumes constant prompt-dependent evidence in its main experiments. What evidence supports the framework's ability to distinguish latent preferences from prompt-specific effects when the latter vary across prompts? In other words, since the experiments assume constant prompt evidence, how do we know the method can still separate model preferences from prompt effects when the prompts differ?
2 comments

Thank you for the clarifying questions regarding the answer space mapping as well as context sensitivity and prompt-specific evidence. We hope our responses below clarify the modeling choices we made as well as the organizational structure of the BBQ dataset.

Question 1 (Exploration of finer or larger answer spaces): Our {S,A,U} answer space for this experiment comes directly from the original organization of the BBQ dataset and is not an external mapping that we are imposing on the data. We are aligning the already provided answer choices with their semantic roles curated by the dataset.

We do acknowledge however that the exploration of finer and larger answer spaces is an excellent direction for future research. Our current contribution is to demonstrate how our proposed surrogate model can help interpret and further understand already established social bias benchmarks, of which BBQ is a prime example. Unfortunately, BBQ in its current form does not natively support finer/alternative groups and exploring such answer spaces would require an adaptation of this setup along with the use of additional benchmarks.

We do show, however, that our formulation does in fact support finer answer spaces with our shift-cipher experiments having 26 response categories. These experiments demonstrate that our method can be applied to a meaningful extent beyond a small set of 3 response categories as seen in our main BBQ experiment.

Question 2 (Stability of our reconstruction): We have incorporated an additional experiment to demonstrate the stability of our reconstruction to prompt variations. We have updated the manuscript and the new experiments can be found in the new Appendix C. The updated text is written in blue.

As a quick summary, we run our reconstruction algorithm on different random splits of the ambiguous BBQ dataset that contain different prompt variations intended to evaluate the same types of bias. We find that the reconstructed priors maintain stable under different randomized splits with a low mean absolute difference between the probabilities.

Question 3 (Nonconstant prompt evidence setting): Although the main LLM experiments use constant prompt evidence, Appendix B of our original submission explicitly evaluates nonconstant prompt evidence. We generate observations from a known multimodal prior and inject prompt-specific evidence over added Gaussian noise. Figure 4 shows that the reconstructed distribution closely follows the known prior while the empirical mean baseline is influenced by the injected prompt-specific evidence.

Reviewer_bn99OpenReview ↗

Reviewer Response

I appreciate the authors' responses and additional experiments, they address part of my concerns. While some questions regarding its generalizability and applicability to real-world settings remain, I find the revisions helpful. I'll keep my score.

Reviewer_eM3cOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to find the prior distribution of preferences in LLMS such that it's possible to explain their biases. Authors introduce a Bayesian surrogate model to mimic LLM behavior. In this surrogate, the LLM samples first from a discrete latent "bias state" and then generates the output according to this.

Critical strengths
  • The paper asks an interesting question: what can we learn about a model’s response preferences beyond a single bias score? The proposed approach yields a full distribution, giving a more detailed view of the behavior.
  • The synthetic experiment makes sense as a sanity check for the method.
  • The mathematical setup seems reasonable, and it helps separate the preferences of the model across prompts and specific to prompts.
  • I appreciate the author's clear writing and presentation of the paper.
Critical weaknesses
  • The main RQ of the paper, as stated in the intro is "Can we reconstruct a latent prior distribution over possible biased responses that explains an LLM’s stereotypical preferences?" However, as the authors noted, we can not assume that LLMs are Bayesian, nor does the surrogate model represent an explicit internal object in the LLM. Thus, it's unclear whether the explanations obtained via the authors method are actually useful for explaining the LLM's preferences. I.e., a proper explanation should be able to provide causal evidence of which factors are affecting the behavior, and provide good predictions in this sense.
  • Section 6.2 mentions that the inferred priors are well identified due to the credible intervals being relatively tight. However, due to the chosen loss function, these intervals could become narrow just because there are enough examples, even if the inferred prior partly reflects how the prompts are worded rather than a stable preference of the model. Tight intervals alone thus do not show that the recovered preferences are stable across different prompts.

Not critical, but worth mentioning:

  • The linear model that authors propose (with Gaussian noise and Gaussian prior), has an exact Gaussian posterior, which has a unique optimum that can be computed exactly (no need thus for gradient descent nor Laplace approximation)
  • Probably a presentation issue, but the Algorithm 1 sums the data loss only over the minibatch, while the negative log-posterior term remains fixed, making it compound over mini batches.
Questions for authors
  • Are the preferences learnt by the surrogate model actually causal for the LLMs being analyzed?
  • What happens if they don't have the preferences that the surrogate points to?
  • Have you checked how well the surrogate fits the observed responses, and whether the inferred priors remain stable across meaning-preserving prompt variations (e.g., paraphrasing)?
2 comments

Thank you for the questions regarding the interpretation of the surrogate model and the stability of our reconstruction across prompt variation. We hope the below responses clarify the scope of the surrogate model and address your concerns about prompt sensitivity.

Question 1 (Causality of our model): Our goal is to provide a statistical characterization of LLM behavior and in particular we do not establish that the reconstructed preferences are actual causal internal mechanisms. As described in Section 4.1, the inferred prior is a low-dimensional surrogate parameter, not an explicit object assumed to be stored within the LLM. Its usefulness lies in expressing the response tendencies of these models as interpretable preference distributions, enabling comparisons across models and bias dimensions. We do not claim that these distributions are explicitly observed in the LLM’s weights or activations, or that they identify causal internal mechanisms.

Analogously to our approach, Bayesian models in human cognitive science can characterize behavioral patterns without requiring or expecting the underlying system to implement explicit Bayesian updates. Our framework allows us to extract some information to further explain the response tendencies of a specific model under a specific task domain.

Question 2 (What happens when the bias does not lie in the proposed categories): While we are specifying our surrogate model to analyze whether or not the LLM has biases in certain preference categories, we can still observe whether or not the LLM is indifferent to these preferences. For instance, in the case of our BBQ dataset, because (U) is the correct response in ambiguous BBQ contexts, a preference for (U) and equal weight over (S) and (A) will be expected of an LLM that does not actually have preferences for any of the categories that our surrogate is trying to model.

Question 3 (Stability of our reconstruction): In Appendix B of the original submission, we report the recovery of a known prior in a synthetic experiment in which we show that our method’s prior reconstruction much more closely matches the known prior than a simple averaged probabilities model.

In order to address your question about stability across prompt variations, we have updated the manuscript with a new experiment that can be found in the new Appendix C. The updated text is written in blue.

As a quick summary, we run our reconstruction algorithm on different random splits of the ambiguous BBQ dataset that contain different prompt variations intended to evaluate the same types of bias. We find that the reconstructed priors maintain stable under different randomized splits with a low mean absolute difference between the probabilities.

I appreciate the authors response and clarifications. They address part of my concerns, but I'm still unconvinced on the usefulness of the technique given that they can not be used (at least not yet) as causal evidence of the model's behavior. I'll keep my score.

Reviewer_7EWwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a Bayesian framework for examining social bias in language models. Rather than reporting only the usual benchmark scores, the authors estimate a probability distribution over the model’s responses. They formulate this as an inverse problem, obtain a MAP estimate, and use a Laplace approximation to quantify uncertainty. They test the method first on synthetic data and a shift-cipher task, then on ambiguous questions from six dimensions of the BBQ benchmark using Gemma 3 and Llama 3.2 models. In the BBQ experiments, responses are grouped as stereotype-consistent, stereotype-inconsistent, or unknown. The main finding is that similar BBQ bias scores can hide quite different response patterns, especially differences in a model’s willingness to answer when the context is ambiguous.

Critical strengths

The most useful part of the paper is the distinction it makes between directional stereotyping and answering confidently when the evidence is insufficient. The comparisons across age, disability status, and religion show why a single bias score can be misleading. This is a meaningful observation for anyone trying to evaluate model behavior beyond aggregate metrics.

I also liked that the authors started with a synthetic control and a simpler cipher task before moving to BBQ. That progression makes the method easier to understand, and the mathematical setup is presented clearly.

Critical weaknesses

My main concern is the interpretation of the reconstructed distribution as a latent prior. In the reported experiments, the prompt-dependent evidence term is treated as constant. With that assumption, I am not convinced the method identifies something distinct from the model’s average answer preferences. Since distinguishing prior effects from prompt effects is central to the motivation, this seems like an important limitation rather than just a missing extension.

I also did not see a convincing demonstration that Bayesian reconstruction improves on a simpler baseline. With the linear forward model and Gaussian assumptions used here, the posterior has a closed-form solution. The need for gradient-based optimization and a Laplace approximation is therefore unclear. A comparison against averaging observed log-ratios would make the contribution easier to judge.

Questions for authors
  1. How stable are the reconstructed distributions if the wording or context of BBQ questions changes? Would the same conclusions hold when the prompt-dependent evidence term is not constant?

  2. Given the linear Gaussian formulation, why use gradient-based optimization and a Laplace approximation rather than the exact posterior?

  3. How does the approach compare with a baseline that averages the observed log-ratios before converting them to probabilities, using the same assumptions about prompt effects?

1 comment

Thank you for your careful assessment and your constructive feedback. Below we answer your questions regarding the sensitivity of our model to context changes, the reason for a gradient-based optimization, and how our approach compares to a mean probability estimate.

Question 1 (Stability of reconstructed priors): In order to address your question about stability across prompt variations, we have updated the manuscript with a new experiment that can be found in the new Appendix C. The updated text is written in blue.

As a quick summary, we run our reconstruction algorithm on different random splits of the ambiguous BBQ dataset that contain different prompt variations intended to evaluate the same types of bias. We find that the reconstructed priors maintain stable under different randomized splits with a low mean absolute difference between the probabilities.

To address your second question, we set the relative evidence offsets to zero in this specific experiment because it is motivated by the inherent structure of the ambiguous subset of the BBQ dataset on which we are testing our reconstruction model. The ambiguous subset of the BBQ dataset is curated specifically so that there is no prompt dependent evidence targeting any specific answer choice.

Question 2 (Why use gradient optimization): Our aim for this paper is to present a general computational method in order to support future development on top of this work. For our experiments, we choose to adopt Gaussian prior and noise distributions as modeling assumptions for simplicity. The optimization-based formulation supports future extensions such as learned non-Gaussian priors or nonlinear forward models with datasets that are not specifically curated to have constant prompt evidence. In these cases, closed-form inference may no longer be available.

In this paper, we test this optimization-based formulation on BBQ in a simple setting where we can assume constant prompt evidence providing evidence that our model provides viable results in a controlled setting. We agree that the posterior over the log-ratio parameters in our current formulation admits a closed-form solution and we will clarify this in the revised manuscript.

Question 3 (Comparison of reconstruction to baseline): Appendix B of our original submission compares our reconstruction method against a naive empirical mean. This synthetic experiment validates the importance of our method in accounting for prompt effects that might exist in non-specifically curated datasets. We observe that our method’s prior reconstruction much more closely matches the known prior than a simple averaged probabilities model.

Comments on the paper

1 comment
Area_Chair_ehLsOpenReview ↗

Discussion Period - no rebuttal yet

Dear Authors,

We have not yet seen a rebuttal from you. If you were considering it, please do so as soon as possible to give the reviewers a chance to respond.

Best Regards,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.