acceptodds
Under review as a conference paper at ICLR 2027

Towards Unsupervised Removal of Biased Reasoning in LLM and Humans Alike

Abstract

Rational agents should change their beliefs in response to new information. Still, the martingale property of Bayesian beliefs says that the direction of these changes should not be predictable from their initial beliefs alone. We use the Martingale Score (MS), which measures this predictability, to introduce Martingale Training (MT), an unsupervised training signal, and find that it shows promise. On a forecasting model deliberately distilled with confirmation bias, unsupervised MT improves the held-out Brier score from 0.291 to 0.234 on Qwen3-32B and 0.364 to 0.257 on Llama.3.1-8B roughly matching supervised interventions (direct Brier-score training) which took Brier score from 0.364 to 0.246 on Llama 3.1-8B. In simulated human–AI conversations, MS detects when a sycophantic chatbot entrenches its users, an effect an order of magnitude larger than any entrenchment in the chatbot's own beliefs. Training the chatbot on its user’s MS improves the user’s forecasts and reduces entrenchment, but not better than a simple prompt baseline. As language models increasingly shape what people believe, methods that detect and reduce entrenchment could help build assistants that improve reasoning and human rationality, rather than reinforce existing views and beliefs.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Bayesian Rationality, Human Empowerment, Epistemics, Belief Entrenchment, Confirmation Bias, Martingale

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_crhsOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Martingale Training based on Martingale Scores to repair biased agent's reasoning. Through their experiments, the authors show that punishing bias-distilled LLMs for predictable belief updates via RL improves their belief calibration. In the meanwhile, the authors measure sycophantic influence in human-AI interaction by extending Martingale Score to multi-agent contexts. Finally, the authors demonstrate that Martingale Training can improve user Martingale Scores in human-AI interactions.

Critical strengths

The paper studies belief updating in LLM agents from the perspective of the martingale property. It is important to study whether the LLMs can entrench user beliefs, and how to measure it without labels. Therefore, the paper proposes useful methods, with solid theoretical grounding, to address these questions.

Critical weaknesses
  1. Although useful, there is limited novelty regarding the methods presented in the paper. Previous work has already extensively studied training LLMs to be more Bayesian or non-Bayesian belief updaters through reinforcement learning, and use similar metrics as rewards based on the posterior beliefs and prior beliefs. I think there are nothing fundamentally novel in the paper.
  2. The empirical experiments presented in the paper are also limited in terms of generalizability. Martingale Scores usually need structure that many natural interactions do not have. Real conversations rarely provide a well-defined decision-making context, and the paper shows very limited evidence regarding how the findings can generalize to other settings, for example, with richer action spaces, state spaces, or different domains.
  3. Also, it still remains very questionable regarding the downstream effects of the paper. Although the paper studies some human-AI interactions in the simulated world, the environment is super gameable, and the paper does not provide sufficient human validations regarding the environments. Bayesian agents in the real world are not always the right ideal for assistants. Without considerate discussions regarding the potential effects, it is really hard to appreciate the significance of the work.
Questions for authors
  1. Can you provide further analysis on the relationships between Martingale Scores and Brier Scores observed in the experiments? What do they imply regarding the belief updating behavior of agents?
Reviewer_88zhOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

With increasing use of LLMs and interactions between AI and human users, one important question is about how beliefs are updated, which concerns both the update of human beliefs influenced by LLMs and the own belief updating process of LLMs themselves. Facing such question, this paper is mainly abut evaluating and mitigating the biased belief updating in related cases. To begin with, the papers builds on the martingale score (MS), which is an empirical way to estimate the linear relation between prior belief and belief change (due to conversation and new information). Such MS can be calculated using belief trajectories over given propositions and motivates a training objective intended to reduce predictable belief changes, where the designed reward is the negative multiplication between predicted and measured belief change. With such rewarding scheme, they propose the Product-Based Martingale Training (MT, Algorithm 1). First, by applying MT to a bias-distilled model itself, results show that Brier score has been improved, comparable to supervised training. Second, when interacting with a provided sycophantic instructor modell, the performances of simulated humans (LLMs) drop. Finally, still in the instructor and simulated human setting, when using MT on sycophantic instructor with reward from the simulated human side, simulated humans' accuracy can still be improved, but still worse than simply using de-biasing instructor prompt.

Critical strengths

Interesting question

  • This paper is concerned about an interesting and important question. With LLMs involved in decision processes, the mechanism of belief updating processes deserve investigations.
  • The paper concerns the belief updating processes of both LLMs and the users, which provides a broader perspective on such discussion.
Critical weaknesses

More justification of the interpretation of MS and the training objective

  • From the beginning of the paper, the martingale property has been used to help interpret the nonzero MS as signals of reasoning failures. However, a further discussion on whether assumptions behind such property have been satisfied would be helpful, especially concerning the samples of conversations.
  • In addition, the paper initially associates nonzero MS with failures, while in some numerical results negative MS appear with better performances. Some further discussion on such discrepancy could be necessary.
  • In the last part of the experiments, a simple de-biasing prompt approach (without MT) seems to have better performances than applying MT on sycophantic instructor; and MT provides rather limited benefit for the de-biasing case. The reviewer wonders about the advantage of considering MT in the interaction case.

Insufficient explanation of experimental steps and results

  • In Section 3.3 and Algorithm 1, several notations may benefit from explicit definitions or more explanations.
  • Further details and related formula about the information term in the training can help readers better understand the method.
  • In Section 4.1, the observations of modern models can benefit from more discussion and summary of experimental results.
Questions for authors

For clarification

  • Could the authors discuss whether the assumption behind the martingale property hold in related experimental cases and how that can further help with the interpretation of MS?
  • In the final interaction setting, could the authors clarify the advantage of MT?
  • Could the authors provide further discussion of the information term in the algorithm?
Reviewer_SYUDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper converts the Martingale Score of He et al. [1] into a reinforcement-learning reward that needs no labels. The score measures how much a model's belief updates depend on its prior belief. An LLM judge reads the model's belief at the middle and at the end of its chain of thought. The reward is the negative product of each update and its prediction from the prior by a per-batch regression, combined with an information term and a KL term. On a Qwen3-32B forecaster distilled from confirmation-biased reasoning, training lowers the held-out Brier score, and on Llama-3.1-8B it nearly matches supervised Brier training. In simulated conversations where LLMs play the users, a sycophantic chatbot raises the user's Martingale Score, and training the chatbot on that score lowers the user's Brier score, though less than a de-biasing prompt alone.

References
[1] Z. He, T. Qiu, H. Shirado, and M. Sap. Martingale Score: An Unsupervised Metric for Bayesian Rationality in LLM Reasoning. NeurIPS 2025. arXiv:2512.02914.

Critical strengths

The paper claims a training signal without labels that lowers the Brier score about as well as supervised Brier training, a claim that would matter if it held. For the reasons in Critical Weaknesses, I cannot count this training signal as a separable contribution.

Critical weaknesses

The reward does not lower the Martingale Score it is built on, and Algorithm 1 does not optimize the objective the paper states, so nothing in the paper shows that the proposed idea causes the reported gain.

1. The paper's one new idea, training on the Martingale Score, is not shown to work, so the paper adds little beyond [1]. [1] introduces the score and its interpretation and already proposes training on it as future work, with a test: such training should lower both the Martingale Score and the Brier score. On the distilled model, the runs without labels lower only the Brier score. The Martingale Score barely changes on Llama-3.1-8B and rises on Qwen3-32B (Table 1), and it falls only in runs that use the supervised Brier reward. So the training fails the test of [1]: the reward does not lower the score it is built on.

No run removes the martingale reward or replaces it with a shuffled or random reward while keeping the information, KL and length terms, so those terms may cause the Brier gain instead. No run compares the product reward with a simpler form, such as a direct penalty on the predicted part of the update, and the paper does not say why it chose the product over such a form. Without these controls, the results cannot be attributed to the martingale reward and are hard to trust.

2. The training objective is not what the paper says it is, and the manuscript was not edited for outside readers. Section 3.3 states that maximizing the reward "is equivalent to minimizing MS", the slope $\hat{\beta}_1$ of the regression of belief updates on priors. Minimizing MS is an objective on $\hat{\beta}_1$ alone, and Algorithm 1 optimizes a different quantity. Its reward is minus the product of each update $\Delta_i$ and the update $\hat{\Delta}_i$ that the regression predicts from its prior. Over a batch these products sum to $\sum_i \hat{\Delta}_i^2$, because a regression's residuals are uncorrelated with its predictions. Each $\hat{\Delta}_i$ is the mean update plus $\hat{\beta}_1$ times the prior's distance from the mean prior, so the batch reward is $-n(\bar{\Delta}^2 + \hat{\beta}_1^2 \operatorname{Var}(b_{\text{prior}}))$, where $\bar{\Delta}$ is the mean update. Unlike an objective on $\hat{\beta}_1$ alone, this reward has three effects:

  • It penalizes a negative $\hat{\beta}_1$ as much as a positive one, so maximizing it moves MS toward zero, although Section 5.1 says that rational updating by a systematically overconfident agent gives a negative MS, and footnote 1 describes the distilled model as overconfident.
  • It penalizes the mean update $\bar{\Delta}$, which MS does not involve.
  • It increases when the priors are less spread out, even if $\hat{\beta}_1$ stays the same. Section 3.3 adds the information term so that the model cannot satisfy the objective "by never updating its belief from the prior", but no term in the paper guards against the route that the third effect opens: the model can raise the reward by stating less spread-out priors, with MS unchanged.

The information term is not defined consistently. Algorithm 1 adds it to the loss as a function of the sampled updates, which do not depend differentiably on the model, so it contributes no gradient and does not affect training. Appendix A instead includes it in the reward as a penalty, where it does affect training, so the two descriptions train different objectives. The KL reference model for the distilled runs is not named. If it is the undistilled model, the KL term pulls the distilled model back toward it, which alone could restore calibration. Algorithm 1 also marks PPO clipping and a length penalty as optional, and apart from the length penalty in the two-agent training (Section 3.4), the paper does not say which each run used, so the objective each run trained is unknown.

The manuscript was also not edited for outside readers:

  • The data splits are given as "held-out splits as noted per run (run-ids in the project log)" (Appendix A).
  • Code names appear beside or in place of definitions, such as BRIER_FROM_STATED (Appendix A) and "INSTR_SYC instruction" (Table 2 caption).
  • Labels appear without introduction: "C1 base models" and "C3 instructor" with no C2, "the s0 mean stated-p" (footnote 1), and "(training seed)" attached to both a judge model and a prompt.
  • The top panel of Figure 2 is titled "(Mistral-7B base)", a model used in no experiment the paper describes.
  • The bottom row of Figure 2 is a raster image of four raw diagnostic plots (MS_coef, meanP, options[0]) whose text cannot be read at print size.
  • The same row calls the distilled model "B", the letter Table 1 uses for the Brier reward.
  • Section 3.4 gives settings as command-line defaults, "(default: deepseek)" and "(default: 3)".
  • One model is called "deepseek", "deepseek-chat", "deepseek-chat-v3" and "DeepSeek".
  • Text in Figure 1 is also too small to read at print size.
  • The lower quarter of page 6 is blank.

The related work cites [2] for "under what conditions sycophantic behavior arises" (page 2), but [2] studies models' reluctance to express uncertainty and does not mention sycophancy. The AI use statement says that "references were compiled and checked manually, and all text underwent extensive manual editing and review" (page 8). The listed items and this citation error are each minor, but together they contradict that statement and show that the reported numbers were not checked closely enough to be trusted.

References
[1] Z. He, T. Qiu, H. Shirado, and M. Sap. Martingale Score: An Unsupervised Metric for Bayesian Rationality in LLM Reasoning. NeurIPS 2025. arXiv:2512.02914.
[2] K. Zhou, J. D. Hwang, X. Ren, and M. Sap. Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertainty. ACL 2024.

Questions for authors
  1. Could you report runs on the distilled model without the martingale reward, and with it replaced by a shuffled or random reward, keeping the other terms? If the Brier gain remains, the martingale reward does not cause it. If it disappears, the runs answer Weakness 1, and I would raise my Informativeness rating.
  2. Which model is the KL reference in the distilled runs? If it is the undistilled model, the ablation above would also need to rule out the KL term.
  3. How many seeds does the Qwen3-32B row of Table 1 use, and what is its spread? The paper's main numbers are in this row, and on Llama-3.1-8B the difference from supervised training is already smaller than the spread Table 1 reports, so the spread decides whether the comparisons show a real difference.
Reviewer_BnyXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes martingale training as a method for removing belief entrenchment tendency and confirmation bias in LLMs.

Critical strengths

This is a well-written, well-scoped paper with a very timely and interesting topic. Indeed, the authors are (to my knowledge) first to test a "domain-agnostic, weight-level training regime" to improve Martingale Scores. Using Martingale is a creative idea. The paper's data is fairly strong and (on their metric, see below for a discussion on this) shows a strong gain under the training regimen.

Critical weaknesses

As the authors note on page 7: "Effective belief elicitation is an open question" This draws into question some of the other claims the authors make in the paper, e.g. the claim that their method "repairs biased agents’ reasoning" (pg 2). Another question is whether gains from their method translate into practical circumstances. The paper highlights that, when testing on a chatbot session, the model performs no better than prompt-based methods. (It is commendable that the authors say this so clearly!) Prompt-based methods obviously do not deeply change the model's capabilities for reasoning.

Another related concern is Goodhart's law: "When a measure becomes a target, it ceases to be a good measure." How do the authors ensure that there is no reward hacking or that Martingale Scores even stay informative under RL? It would be better to have a more thorough and diverse evaluation of the model under training.

Questions for authors

How do the authors ensure that there is no reward hacking or that Martingale Scores even stay informative under RL? Do they have data on this?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.