acceptodds
Withdrawn from ICLR 2027

Confidence Without Control: Revision in Language Models Is Confidence-Blind

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

When a user pushes back on a correct answer, language models often give it up. Qwen2.5-7B answers of our held-out questions correctly, and after one sentence, "Actually, another source says the answer is X," its accuracy is . We show that this failure is confidence-blind revision: changing the model's stated confidence, with its answer held fixed, does not change whether it revises. The models know which answers are likely wrong, since stated confidence, P(True), and belief (the log-probability of the answer) predict their correctness with AUROC up to . Yet a perfect confidence note written into the model's own turn ( if right, if wrong) changes accuracy after the challenge by at most in three models from two families, although the models read the note and recall it on at least of items. A wrong-for-wrong test, which challenges a wrong answer with a different wrong answer, shows that Qwen2.5-7B and Llama-3.1-8B switch to the option they believe least as readily as to their second choice. Qwen2.5-14B, Qwen2.5-32B, and Qwen3-8B compare the two options more, and two frontier models abandon far fewer correct answers and often move to the true one. Training makes revision confidence-aware: trained models given a perfect note keep to of their answers correct under challenge, and models trained to follow a confidence threshold realize to of the best achievable gain (three seeds) for notes with AUROC of or more. Ignoring confidence costs accuracy in two ways. By default, models that are usually right abandon answers under an uninformative challenge, and a threshold rule on their own calibrated confidence, which keeps every answer, beats them by up to points. Item by item, a note with AUROC , acted on as well as possible, would leave Qwen2.5-7B at under challenge (it ends at ), but the models' self-report (AUROC to on these items) is too weak to add more than a point beyond keeping every answer. Calibration research should test whether models act on what they know.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
sycophancy, large language models, answer revision, LLM calibration, confidence calibration, preference optimization, RLHF, self-correction, prompt injection, robustness to adversarial prompts, model behavior, epistemic behavior, AI safety, alignment, behavioral evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_eAmKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper's main claim is that LLMs revise their answer when challenged irrespective of their confidence in the initial answer. The authors propose new tests to study how models use confidence when being challenged (perfect-note test and a wrong-for-wrong test). The authors propose a LORA fine-tuning setup to encourage the model to use confidence when challenged. Experiments with models from two different families and questions from three sources are included.

Critical strengths
  • The paper addresses an important topic: LLM confidence and how it drives behaviour under challenge.
  • The Wrong-for-Wrong Test to disentangle self-doubt from active option comparison is interesting.
Critical weaknesses

The paper is difficult to read, probably heavily written with an LLM. Eg. Fig 1 is very hard to parse for me.

The methodology has significant flaws:

  • The confidence note inserted in the model’s turn is basically ignored by the models, but the authors consider this a causal intervention that proves the models don’t take confidence into account when deciding to revise. A proper causal intervention would have been to change confidence internally and see the effects.
  • The training with LORA so the model takes into account the confidence related message leads to non-negligeable decrease in overall model performance. This doesn’t seem like a useful course of action.
  • The experimental protocol seems to change across sections, making it hard to understand the results; in some parts a binary task is used, then for MMLU, a ten-options with open answer; then an “earlier design” with modified text prompts
  • Multiple pre-registered predictions were not met or only partially met (Table 11), which points to weaknesses in initial hypotheses.
Questions for authors

See Weaknesses above.

Reviewer_JqEuOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper works on confidence of LLMs.

Critical strengths
  1. It's written by language.
  2. It provides codes.
Critical weaknesses
  1. I can not understand what's the point of this paper.
  2. I don't know the meaning of the metrics in experiments. It seems that they are some AI-generated concepts.
  3. Academic writting regulations are not performed in this paper.
Questions for authors

nan

Reviewer_XYe5OpenReview ↗
Rating
1
Confidence
1
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how models respond to user pressure, as a function of what their verbalized confidence is in their original answer. Using prefills on open-weight models, experiments elicit verbalized confidence from models and then adjust the stated verbal confidence scores -- referred to in the paper as "confidence notes". Experiments show that swapping stated confidence between 95% and 5% has negligible effect on how often models switch their answer in response to follow-up pressure from users in dialogue to switch their answer. Next, the paper introduces training experiments for improving the sensitivity of model answer switches to their verbalized confidence scores. But the paper suggests that, while training does help ground answer switching in stated confidence, this comes with costs to first-answer accuracy.

Critical strengths
  • The paper addresses the important problem of calibration for LLMs, especially looking at verbalized confidence. It studies an interesting intersection of calibration and sycophancy, or caving to user pressure to switch answers in dialogue.
Critical weaknesses
  • This paper was not ready for submission to ICLR. The presentation needs to be sustantially improved in order to make it legible to a human. Many sentences lack basic definitions, referents, and context, starting with the first one in the paper: "Ask an open-weight 7B model an easy science question and its accuracy exceeds 0.97." What is an easy science question? How is accuracy measured in a fractional way on a single question? What kind of open-weight 7B model? The paper presentation makes it extremely difficult for me to parse the key results and contributions of the paper. It is really hard to understate this. Even looking to the main conclusions of the paper, it is hard to tell what to take away. What does this snippet from the conclusion section, for example, mean: "Language models can tell when they are likely wrong, but when a user pushes back they do not use that knowledge to decide whether to keep their answer. A perfect confidence note in their own words, which they can recall, leaves Qwen2.5-7B at 0.230." There are too many possible interpretations of this sentence, many of which would conflict with previous conclusions from related, cited papers in the submission, to accurately assess the validity of the paper.
  • Key definitions and assertions, where included at all, need to be defended or better motivated. What makes a confidence note "perfect"? Why should the verbalized confidence by 95% when the model is correct and 5% when it is incorrect. These are arbitrary choices that ignore underlying conceptual challenges, such as distinguishing between the confidence we have in an answer and the confidence that a model has in an answer.
Questions for authors
  • see weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.