acceptodds
Withdrawn from ICLR 2027

Self-Blinding and Counterfactual Self-Simulation Mitigate Biases and Sycophancy in LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Fair decisions require ignoring irrelevant, potentially biasing, information: a decision-maker must approximate what they would have decided had they not known, say, a job candidate's gender or race. This counterfactual self-simulation is notoriously hard for humans, producing biased judgments even by well-meaning actors. We show that large language models (LLMs) share this limitation, in offsetting gender and race biases and in overcoming sycophancy: prompting models to ignore or pretend not to know biasing information fails – and occasionally backfires. Unlike humans, however, LLMs can be given a ground-truth model of their own counterfactual cognition: their own API. Enabling models to query blinded copies of themselves yields fairer decisions and an auditable record of every departure from the blinded verdict.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
sycophancy, bias, fairness, tool use, counterfactual reasoning, LLMs, evaluations

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1jV3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes self-blinding, where an LLM sends a bias-filtered prompt to a fresh copy of itself and gets the answer, to remove race, gender, and user-identity bias. Prompting the model to ignore the biasing information fails across models, and only Claude Opus 5 makes the full self-blinding loop work end to end on both tasks.

Critical strengths

A simple idea: querying a copy of yourself that knows less. Very honest reporting of failures and confounds, and strong reproducibility.

Critical weaknesses

W1 The method is very basic and not novel at all. It is a well-established method under fairness via unawareness, which has a lot of limitations, such as removing the sensitive attribute does not necessarily remove bias, as it may be learned from other shortcuts. Also, the debiasing can be easily overridden by providing other prompts to the model.

W2. Related work section is very thin and has missed very close prior work, For example, two simple Google searches find:

a. sycophancy third-person perspective reframing reduces sycophancy LLM returns "Ask Don't Tell" (arXiv 2602.23971). Third-person framing already reduces sycophancy.

b. LLM position bias swap order calibration pairwise evaluation "Judging the Judges" (arXiv 2406.07791), which build on “Large Language Models are not Fair Evaluators” Wang et al. (ACL 2024).

The paper need to cover at least methods based on fairness via unawareness and prompt-based approaches for debiasing, where both methods are highly saturated and hard to bring novel ideas. The opening of the introduction (paragraph 2, 3) is basic and the state of the field from AI and debiasing perspective is most important missing to situate the work in the existing litrature.

W3. The results do not support the hypothesis. Only one model fully works on both tasks; Qwen and GPT-4.1 are not mitigated end-to-end. Gemini improves on names but not on disputes. Only Opus 5 succeeds on both.

W4. Gemini's bias falls, although only 39% of its trials contain a blinded call, and GPT-4.1 changes with the tool merely present. There is no control with the tool present but its output ignored. So, the mechanism is not proven.

W5. Please clarify what the error bars are demonstrating? We often use them to show std or 95% CIs, but the definition in the paper do not reflect them. Deferral, invoke, and compose rates and the end-to-end coefficients have no uncertainty estimate. There is no correction for multiple comparisons.

Questions for authors

The paper in its current form is more challenging to accept than it is to improve during rebuttal. It requires major modification. I have the following suggestions to improve the paper quality for future.

Add two controls: the tool present but its output hidden, and a random verdict supplied in place of the blinded one. These separate self-blinding from tool presence and from obedience to any tool output.

Report 95% confidence intervals for every headline number, including deferral, invoke, and compose rates and the end-to-end coefficients, with a correction for multiple comparisons.

Make the related work stronger to better situate the novely of the work within the existing AI literature and position the novelty.

Also, cover third-person reframing and position-bias calibration, add the in-context third-person rewrite as a baseline, and narrow the claims to what Opus 5 shows.

Reviewer_vG2AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper first shows that simple prompt tuning using the de-bias instruction can be less effective and worse in some cases. The authors then propose to use a self-blindness method to avoid it by calling a tool run_counterfactual_simulation to construct a blinded prompt by removing potentially biasing information. Removing context is a crutial problem in fairness, and it also an important ability of instruction following. Experimental results show that this approach can substantially reduce demographic biases and sycophancy compared to the prompt instruction based approach.

Critical strengths
  1. The paper studies an important problem: LLMs may not be able to reliably ignore information that is already present in their context, even when explicitly instructed.
  2. The experiments are solid and thorough.
  3. Paper writing is easy to follow.
Critical weaknesses
  1. My major concern is that I am not fully convinced by the methodological novelty of the proposed self-blinding approach. For example, System 2 Attention (Weston & Sukhbaatar, 2023) and BiasBuster (Echterhoff et al., 2024) have already proposed rewriting the original context to remove irrelevant or opinionated information and then querying an LLM using only the rewritten context. The central mechanism of separating the relevant information from the original context and making a new inference call is therefore quite similar. This paper, to some extent, is an agentic implementation of S2A, which is quite incremental.

  2. Another key challenge is that we cannot simply construct a de-biased version x′x' of the original input xx and call the model again. The more difficult problem is how we can reliably construct such an x′x' while preserving the information necessary for the original decision, rather than just trusting the agentic tool to do so. Of course, I agree that we can rely on an out-of-the-box LLM to solve this problem, and we can conduct some evaluations (like the counterbalanced evaluation in the paper). But if an out-of-the-box LLM's ability to construct counterfactual inputs is already good enough, I still see this paper as incremental. Otherwise, improving the LLM's ability in this regard seems to be a more interesting problem. If the creative contribution is to discover and evaluate the ability to generate counterfactual inputs, this has already been explored by Li et al. (2024).

  3. Another concern is that selectively deleting information from the context is neither generally sufficient nor necessary for fairness. For harder counterfactual fairness cases, deleting information itself is not sufficient. Thus, the connection between the proposed mechanism and the broader fairness problem is not sufficiently developed (I understand that the authors mentioned this in the paper, but I think this raises a discussion I want to include here). The self-blinding approach might also be applicable to tasks beyond fairness. The ability to forget is an important general ability of LLMs. For example, in RAG systems, can an LLM reliably ignore irrelevant or misleading retrieved documents? In multi-turn conversations, can a model ignore historical information that is not directly relevant but has spurious correlations with the current task?

  4. There is another line of research on selective attention and in-context forgetting (Weston & Sukhbaatar, 2023; Qian et al., 2026). I suspect that a well-designed and executed self-blinding pipeline could serve as a useful performance reference for in-context forgetting approaches (soft self-blinding). Moreover, Yu et al. (2024) have already shown that S2A can be distilled into a single-pass model without extra inference calls. This raises the question of whether the proposed self-blinding approach could provide a better teacher for improving a student model's ability to selectively ignore biasing information, compared to existing S2A distillation.

References

Weston, J., & Sukhbaatar, S. (2023). System 2 Attention (is something you might need too). arXiv preprint arXiv:2311.11829. https://arxiv.org/abs/2311.11829

Echterhoff, J. M., Liu, Y., Alessa, A., McAuley, J., & He, Z. (2024). Cognitive Bias in Decision-Making with LLMs. Findings of the Association for Computational Linguistics: EMNLP 2024, 12640–12653. https://aclanthology.org/2024.findings-emnlp.739/

Li, Y., Xu, M., Miao, X., Zhou, S., & Qian, T. (2024). Prompting Large Language Models for Counterfactual Generation: An Empirical Study. Proceedings of LREC-COLING 2024, 13201–13221. https://aclanthology.org/2024.lrec-main.1156/

Qian, Y., Yang, Z., Ding, W., Li, H., & Xie, Y. (2026). Do LLMs Forget What They Should? Evaluating In-Context Forgetting in Large Language Models. ICLR 2026. https://openreview.net/forum?id=hcJywRYc3n

Yu, P., Xu, J., Weston, J., & Kulikov, I. (2024). Distilling System 2 into System 1. NeurIPS 2024 Workshop on System 2 Reasoning. https://arxiv.org/abs/2407.06023

Questions for authors

N/A

Reviewer_ULrgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper aims to identify (and mitigate) potential bias exposed by LLM using self-blinded counterfactual reasoning. Instead of prompt LLM to "ignore some information", the authors proposes a tool-use self-blinding method which gives an LLM API access to a fresh-context copy of itself. By spawning a fresh context window with the biasing details redacted or swapped, a model can query its truly "blinded" self rather than guessing. The authors built three controlled counterfactual datasets about demographic bias, sycophancy, and name bias. On these datasets, they notice the limitation of prompting-based mitigations and self-blinding method can reduce demographic dias and exposes residual sycophancy.

Critical strengths

The idea of self-blinding by a separate API call with "sanitized" context is intuitive and novel. The author also made a good comparison to contrast "trying to ignore something you already read" (usually fails) with "asking a fresh copy of yourself who never saw it" (and works better).

The idea of wrapping self-blindness into a tool call is clean and agentic. In the ideal case, a fair LLM should proactively identify the irrelevant information in its prompt and decide to invoke a tool and write the redacted prompt, following by self-reflection / self-evaluation on the response agreement as a mitigation strategy.

The counterfactual datasets they designed also serve as a good resource for future research.

Critical weaknesses

I'm concerning the soundness of their evaluation metrics. For example, ΔPrompted​ is defined as $ R_{Prompted}(unblinded)−R_{Default}(blind) $, instead of $ R_{Prompted}(unblinded)−R_{prompted}(blind) $, so any instruction that uniformly shifts the model’s overall base rate of saying "Yes" can also inflates ∣Δ∣. Or think about an extreme case where the model treats every race and gender 100% identically. This is good, however, ∣Δ∣ can be inflates.

I'm also concerning of whether logit difference is a good proxy. In section 3-5, they measure response in unbounded log-odds, where a large logit change does not necessary mean a decision change. Their Table I.1 is a good example -- when converted to probability space, those large logit biases shrink to just 2–5 percentage point changes.

Also, in the section 4 (dispute) and section 5.1 (pairwise resume), Δ\Delta and ∣Δˉ∣|\bar \Delta| only measure swap invariance. A model that ignores candidate qualifications and flips a 50/50 coin achieves a "perfect" $\Delta =0 $ and ∣Δˉ∣=0|\bar \Delta|=0. In cases where one candidate has an extra concrete achievement, how LLM will behave? Is self-blindness call able to preserve sensitivity to actual merit?

My final concern is about the blindness granularity. Currently, the claimed self-blindness are mostly the Textual Blindness which may not be enough. For example, stripping explicit race/gender (using they/them) does not remove real-world proxy variables (e.g., girl schools, sport clubs, surrounding neighborhoods, dialects, socioeconomic cues) that also provide implicit priors.

Questions for authors

See weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.