- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes self-blinding, where an LLM sends a bias-filtered prompt to a fresh copy of itself and gets the answer, to remove race, gender, and user-identity bias. Prompting the model to ignore the biasing information fails across models, and only Claude Opus 5 makes the full self-blinding loop work end to end on both tasks.
Critical strengths
A simple idea: querying a copy of yourself that knows less. Very honest reporting of failures and confounds, and strong reproducibility.
Critical weaknesses
W1 The method is very basic and not novel at all. It is a well-established method under fairness via unawareness, which has a lot of limitations, such as removing the sensitive attribute does not necessarily remove bias, as it may be learned from other shortcuts. Also, the debiasing can be easily overridden by providing other prompts to the model.
W2. Related work section is very thin and has missed very close prior work, For example, two simple Google searches find:
a. sycophancy third-person perspective reframing reduces sycophancy LLM returns "Ask Don't Tell" (arXiv 2602.23971). Third-person framing already reduces sycophancy.
b. LLM position bias swap order calibration pairwise evaluation "Judging the Judges" (arXiv 2406.07791), which build on “Large Language Models are not Fair Evaluators” Wang et al. (ACL 2024).
The paper need to cover at least methods based on fairness via unawareness and prompt-based approaches for debiasing, where both methods are highly saturated and hard to bring novel ideas. The opening of the introduction (paragraph 2, 3) is basic and the state of the field from AI and debiasing perspective is most important missing to situate the work in the existing litrature.
W3. The results do not support the hypothesis. Only one model fully works on both tasks; Qwen and GPT-4.1 are not mitigated end-to-end. Gemini improves on names but not on disputes. Only Opus 5 succeeds on both.
W4. Gemini's bias falls, although only 39% of its trials contain a blinded call, and GPT-4.1 changes with the tool merely present. There is no control with the tool present but its output ignored. So, the mechanism is not proven.
W5. Please clarify what the error bars are demonstrating? We often use them to show std or 95% CIs, but the definition in the paper do not reflect them. Deferral, invoke, and compose rates and the end-to-end coefficients have no uncertainty estimate. There is no correction for multiple comparisons.
Questions for authors
The paper in its current form is more challenging to accept than it is to improve during rebuttal. It requires major modification. I have the following suggestions to improve the paper quality for future.
Add two controls: the tool present but its output hidden, and a random verdict supplied in place of the blinded one. These separate self-blinding from tool presence and from obedience to any tool output.
Report 95% confidence intervals for every headline number, including deferral, invoke, and compose rates and the end-to-end coefficients, with a correction for multiple comparisons.
Make the related work stronger to better situate the novely of the work within the existing AI literature and position the novelty.
Also, cover third-person reframing and position-bias calibration, add the in-context third-person rewrite as a baseline, and narrow the claims to what Opus 5 shows.