acceptodds
Under review as a conference paper at ICLR 2027

Vision-Language Model Confidence Is Not a Property of the Answer

Abstract

Vision-language models are increasingly deployed behind a confidence gate: the system reads how confident the model is in its answer and defers when that confidence is low. This makes the confidence signal itself worth attacking. We show that a white-box adversary who perturbs only the input image, within an budget of and under the constraint that the model's answer stay byte-identical, can invert the confidence ranking, lowering it on correct answers and raising it on wrong ones until the signal points the wrong way. The answer never changes, and most of the inversion persists even when the whole next-token distribution is held near the clean one, so the answer does not determine the confidence attached to it. Across four vision-language models and three visual question-answering benchmarks, the attack drives the model-internal readouts we test below chance in 83 of 84 readout-by-cell profiles under an adversary that knows which answers are correct, and six of the seven readouts fall below chance in every cell under an adversary that does not. Training a probe on the model's frozen hidden states does not fix this: the robustness such a probe gains against the attack is paid for with the information that made it useful. Nor does reading confidence from a separate verifier model, which holds only until the attacker can reach it. Nor does an adversarially trained vision encoder, which at its own training budget holds against the attacker without correctness labels but not against the one with them. How far an answer-preserving adversary can reach a signal governs where it survives, and whether a robust and informative readout can be built remains open. For deployment, a gate under this attack admits up to of the wrong answers it would otherwise catch, when the attacker has no correctness labels, and, corrected for how often the model is wrong, the answers it accepts are less accurate than with no gate at all.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
vision-language models, confidence estimation, adversarial attacks, uncertainty quantification, multimodal models, model calibration

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_TwMUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper attacks the confidence signal that deployed vision-language models (VLMs) use to decide whether to accept or defer an answer. A white-box attacker perturbs only the input image (L∞ ≤ 8/255) and is required to keep the model's generated answer byte-identical. The attack lowers confidence on correct answers and raises it on wrong ones. Across four VLMs (InternVL3.5-2B, Molmo2-4B, LLaVA-OneVision-7B, Gemma-3-12B), three benchmarks (GQA, VQAv2, POPE) and seven confidence readouts, the authors report:

  • with correctness labels, the attack drives 83 of 84 readout-cell combinations below chance; without labels, six of the seven readouts fall below chance in every cell;
  • 62% of the inversion survives when the full next-token distribution is held within 0.2 nats (KL) of the clean one;
  • attacked AUROC splits exactly into a same-answer term and a cross-answer term, which explains the one surviving case (Molmo/POPE);
  • adversarially trained probes, a robust vision encoder (FARE) and a separate verifier VLM all fail to give a confidence signal that is both informative and robust under a white-box attacker;
  • a gate tuned to a 10% false-accept rate admits up to 89.8% of the wrong answers it would otherwise reject, and, at each model's natural error rate, accepts answers less accurate than having no gate.
Critical strengths
  • The constraint is strict and verified. Every answer token must stay the top prediction during the attack, and the final 8-bit image is re-decoded and must reproduce the answer byte for byte (Section 3).
  • The main attack result is well checked. Within the tested models and readouts, it holds across three attack seeds, at half the budget (4/255), and when each readout is scored only on the attack aimed at it (Section 4).
  • The KL-bound experiment is a convincing check. It rules out the explanation that the attack only exploits low-probability tokens (Section 4).
  • The same-answer / cross-answer split is exact and useful. It explains why Molmo/POPE survives, and the long-answer experiment isolates the cross-answer mechanism (Sections 4–5).
  • The deployment analysis is concrete. The gate experiment, including the combined attack that first plants a wrong answer and then gets it past the gate (77% on LLaVA, 88% on Molmo), shows practical impact (Section 4).
  • The reporting is honest. Success thresholds were fixed before the experiments, and limitations are stated clearly.
Critical weaknesses

1. The novelty is modest.

  • The paper itself states that a confidence attack with the prediction held fixed "is therefore settled" by prior work on image classifiers and text language models (Section 1), and names ACE as the closest predecessor (Section 2).
  • The attack is standard PGD, and all seven readouts are existing methods.
  • The main conceptual contribution is therefore extending a known attack to open-ended VLM generation and showing that common defenses do not stop it.
  • The question that would add the most depth is left open: why one model's confidence can be moved and another's cannot. On long answers, the attack inverts the readouts on Gemma and Molmo but not on InternVL, and none of the properties the authors examine (reasoning training, answer length, backbone, size, strength of the correctness signal, input-to-logit sensitivity) predicts this split (Section 5). The Conclusion lists it as the open question.
  • Even a partial answer would raise the contribution well beyond extending a known attack, for example a hypothesis tested on the existing models, or a controlled comparison between a model whose confidence moves and one whose confidence resists (see Question 6).

2. Model coverage is incomplete.

  • The Qwen-VL family, one of the most widely used open VLM families at this scale, is not tested.
  • The largest model is 12B, and each family is represented by a single model, so the effect of model size cannot be assessed. If compute is the constraint, adding one or two size variants within an existing family (for example, a smaller and a larger model of the same family) would show whether the effect changes with scale.

3. The readouts cover fewer method families than seven suggests.

  • Four of the seven readouts (P(IK), SAPLMA, FULL, COUPLED) are the same MLP architecture on hidden states, and CCPS-D is a differentiable surrogate of CCPS (Appendix A.1).
  • Sampling-based uncertainty (agreement across several sampled answers) is not attacked (Section 2, Limitations).
  • Verbalized confidence appears only in the universal-perturbation control, not in the main attack (Section 3).
  • A gate that combines several readouts is not tested. The paper's results suggest images aimed at one readout already move others, but a joint attack on a combined gate is not evaluated.

The claim that no readout "stays both informative and reliable" (Conclusion) should be limited to single-pass readouts, or these families should be tested.

4. The defenses tested are limited.

  • The model itself is never adversarially fine-tuned; the robustness-versus-informativeness trade-off is shown only for probes on frozen hidden states (Section 5, Limitations).
  • Only one robust encoder (FARE) is tested, on one model (LLaVA).
  • The verifier is described as "a second, independently trained model" (Section 6), but it is one of the other three studied VLMs answering a yes/no prompt (Appendix H.1). A model trained or fine-tuned for verification, a larger model, or several verifiers combined might behave differently.
  • Randomized defenses and adaptive purification are not tested; JPEG purification is evaluated only against a non-adaptive attacker and on three models (Limitations).

5. The threat model narrows practical relevance.

  • The attack is white-box and per-item. It does not transfer across models, and no universal perturbation works (Limitations, Appendix I.1).
  • JPEG at quality 50 removes between a quarter and two thirds of the manipulable range (Limitations).
  • The threat therefore applies mainly to open-weight deployments where the attacker controls the image and has gradient access. The paper should state this scope more prominently in the abstract and introduction.
Questions for authors
  1. Do the results hold for Qwen-VL models and for models larger than 12B?
  2. How do sampling-based and verbalized confidence behave under the same answer-preserving attack?
  3. Does adversarially fine-tuning the full model, rather than a probe on frozen states, change the robustness-versus-informativeness trade-off?
  4. Images aimed at one readout already move others (for example, TokProb inverts without being targeted). Under a joint attack on a gate that combines several readouts, or several verifiers, does the combined gate fail as completely as a single readout does?
  5. Would a verifier trained specifically for answer verification behave differently from the prompted VLMs used here?
  6. Do you have any hypothesis for what makes a model's confidence more or less movable under an answer-preserving perturbation?
16 comments
AuthorsOpenReview ↗

Author Response to Reviewer TwMU (1/2)

We thank Reviewer TwMU for the detailed review and for recognizing the strict, verified answer-preservation constraint, the decomposition, and the honest reporting.

How to read our response. Each row of the table below gives one of your concerns (TwMU.1 to TwMU.10), quoted from your review, with a short, direct answer beside it. The table continues in our next message, which ends with a summary. Detailed replies then follow in this thread in the same order, titled TwMU.1, in detail to TwMU.10, in detail. They provide a more detailed response and point to the corresponding changes in the uploaded revised manuscript, where changes are marked in blue.

Your concernOur response
TwMU.1 “The paper itself states that a confidence attack with the prediction held fixed ‘is therefore settled’”; “The attack is standard PGD, and all seven readouts are existing methods”; “The main conceptual contribution is therefore extending a known attack to open-ended VLM generation and showing that common defenses do not stop it.”The precedent and the attack method are established, but whether confidence-based oversight of a generative VLM can be trusted was not: we show that an attacker can reverse a gate’s decisions without changing any answer, so the gate accepts less accurate answers than no gate at all. We also show what each safeguard does and does not protect, including a new commercial-API attack that inverts confidence without gradients and new joint model–probe adaptation that improves resistance.
TwMU.2 “Why one model’s confidence can be moved and another’s cannot”; “Even a partial answer would raise the contribution well beyond extending a known attack”; “Do you have any hypothesis for what makes a model’s confidence more or less movable under an answer-preserving perturbation?”Our hypothesis is that confidence is harder to manipulate when the image changes that move its readout also threaten the tokens that must be preserved. Two tests support it: our new clean-image estimate, which accounts for answer preservation, ranks achieved movement better than plain sensitivity in six of eight model–probe groups, and holding token margins near their clean values raises attacked AUROC from 0.0000.000–0.1330.133 to 0.5250.525–0.7530.753.
TwMU.3 “The Qwen-VL family […] is not tested”; “The largest model is 12B, and each family is represented by a single model”; “Do the results hold for Qwen-VL models and for models larger than 12B?”Yes: we added Qwen2.5-VL at 3B, 7B, and 32B, and confidence inverts at every size. Clean AUROC of 0.6690.669–0.8430.843 falls to 0.0000.000–0.0780.078, with no consistent gain in resistance as the model grows.
TwMU.4 “The readouts cover fewer method families than seven suggests”; “Sampling-based uncertainty […] is not attacked”; “Verbalized confidence appears only in the universal-perturbation control”; “How do sampling-based and verbalized confidence behave under the same answer-preserving attack?”We revised the paper to separate readouts from method families and to limit our conclusions to what was tested. We also added direct attacks on self-consistency and verbalized confidence, and both fall below chance, with LLaVA’s self-consistency AUROC dropping from 0.8040.804 to 0.0420.042.
TwMU.5 “A gate that combines several readouts is not tested”; “a joint attack on a combined gate is not evaluated”; “Under a joint attack on a gate that combines several readouts […] does the combined gate fail as completely as a single readout does?”We added a direct attack on a fitted combination of token probability, P(IK), and SAPLMA, and the combined score inverts in every setting, with and without correctness labels. Its AUROC falls from 0.7250.725–0.8510.851 to 0.0020.002–0.1650.165 with the answers unchanged.

Our response to TwMU.6 to TwMU.10 continues in the next message.

AuthorsOpenReview ↗

Author Response to Reviewer TwMU (2/2)

Continued from our previous message.

Your concernOur response
TwMU.6 “The model itself is never adversarially fine-tuned”; “the robustness-versus-informativeness trade-off is shown only for probes on frozen hidden states”; “Does adversarially fine-tuning the full model, rather than a probe on frozen states, change the robustness-versus-informativeness trade-off?”We added joint adversarial adaptation of the answering model and the probe, using LoRA across the vision encoder, projector, and language model, and found that it improves the trade-off. Attacked AUROC rises from 0.0650.065 to 0.4650.465–0.5090.509 across two initializations while clean confidence stays informative.
TwMU.7 “Only one robust encoder (FARE) is tested, on one model (LLaVA).”We added TeCoA-4, a second robust vision encoder trained with a different objective, and found the same pattern as with FARE. Both make confidence harder to manipulate, most clearly at small budgets, but neither prevents inversion at the headline 8/2558/255 budget.
TwMU.8 “A model trained or fine-tuned for verification […] or several verifiers combined might behave differently”; “Would a verifier trained specifically for answer verification behave differently from the prompted VLMs used here?”; “Under a joint attack […] on several verifiers, does the combined gate fail as completely as a single readout does?”We added two trained verifiers, a reward model and a critique-and-score evaluator, along with combinations of their scores, and found that a trained verifier can be inverted and that combining verifiers does not prevent it. A direct attack takes the reward model’s AUROC to 0.0000.000–0.0040.004 and the combined score to 0.0000.000–0.0370.037, with the answers unchanged.
TwMU.9 “Randomized defenses and adaptive purification are not tested”; “JPEG purification is evaluated only against a non-adaptive attacker”; “JPEG at quality 50 removes between a quarter and two thirds of the manipulable range.”We added adaptive attacks on JPEG purification and randomized smoothing, and both defenses fail once the attacker accounts for them. Attacked AUROC falls to 0.0360.036–0.1560.156 under JPEG and to 0.1110.111 under smoothing, with the answers unchanged.
TwMU.10 “The threat therefore applies mainly to open-weight deployments where the attacker controls the image and has gradient access”; “The paper should state this scope more prominently in the abstract and introduction.”We added attacks on commercial API models, where GPT-5.4-nano’s AUROC falls from 0.6930.693 to 0.1560.156 using only returned token probabilities, so the vulnerability does not require open weights or gradients. We also revised the abstract, introduction, and threat model to state the scope of the core evaluation, which is white-box and per-image.

Summary. You asked for (1) a contribution beyond extending a known attack, including why confidence is movable, (2) broader model coverage, (3) broader readout coverage, (4) stronger defenses, and (5) a clearer statement of scope. We addressed (1) with a tested hypothesis for what makes confidence movable; (2) with the Qwen2.5-VL family up to 32B; (3) with direct attacks on self-consistency, verbalized confidence, and a combined readout; (4) with joint model–probe adaptation, a second robust encoder, trained and combined verifiers, and adaptive attacks on JPEG and smoothing; and (5) with inversion through a commercial API and a revised statement of the threat model.

We thank you again for the constructive feedback and invite you to read our detailed replies to each of your ten concerns below. We believe they address your concerns and hope you will consider raising your score.

TwMU.1, in detail “The paper itself states that a confidence attack with the prediction held fixed ‘is therefore settled’”; “The attack is standard PGD, and all seven readouts are existing methods”; “The main conceptual contribution is therefore extending a known attack to open-ended VLM generation and showing that common defenses do not stop it.”

Thank you for raising this. We agree that fixed-prediction confidence attacks are established and PGD is standard. We evaluate published estimators alongside an answer-coupled readout and control constructed for this study (Section 3; Appendix A.1). The contribution lies in answering two questions: what happens when a generative system relies on confidence to decide which answers to trust, and what different safeguards actually protect.

First, consider what the gate is supposed to do. It accepts answers judged reliable and sends uncertain ones for review. We show that an attacker can turn this policy against the system without changing the answers: errors are accepted automatically, while correct answers are sent for unnecessary review. Across all twelve core cells at natural error rates, the attacked gate selects less accurate answers than having no gate (Section 4; Appendix F.1, Table 36). We then test the complete adversarial workflow: first create a wrong answer, then keep that wrong answer fixed while raising its confidence. Up to 84% of initially correct items become wrong answers that pass the gate (Section 4; Appendix F.4, Table 39). We thus demonstrate an attack that defeats both the answering model and the check intended to catch its mistake.

The vulnerability also extends beyond short answers: the tested readouts invert across all four models even when the complete answer and its explanation remain exact (Section 5; Appendix C.5, Table 14). We separately test whether the attacker needs (a) correctness labels or (b) model gradients. Without correctness labels, six of seven readouts invert throughout the core matrix (Section 4; Appendix D.2, Table 22). Our new commercial API experiment establishes inversion without gradients: correctness-aware score-only search takes GPT-5.4-nano’s AUROC from 0.6930.693 to 0.1560.156 (Section 4; Appendix I.2, Table 51).

That brings us to the second question: would a better confidence mechanism prevent this? We show that (a) training a probe to resist manipulation on frozen features sacrifices its ability to distinguish errors (Section 5; Appendix G.1, Table 41); (b) our new joint model–probe adaptation improves resistance while keeping clean confidence informative (Section 5; Appendix G.2, Table 42); and (c) an independent verifier helps against transferred attacks, but loses discrimination when directly targeted (Section 6, Table 2; Appendix H, Table 47). These results show why resistance and retained discrimination must be evaluated together, and why an independent check provides protection only under particular attacker-access assumptions.

These findings are more than a catalogue of failed defenses. We establish how confidence-based oversight, intended to make automated answers more reliable, can instead make trust decisions worse. We distinguish what different monitoring designs protect and demonstrate improvements in resistance alongside their costs. TwMU.2 presents the new hypothesis test and intervention addressing your request for greater explanatory depth.

Reviewer_meMCOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates answer-preserving adversarial attacks on confidence estimates in VLMs. The authors show that small image perturbations can manipulate confidence scores without changing the generated answers. Through extensive experiments and analyses, they demonstrate the vulnerability of various confidence readouts and confidence-based gating systems.

Critical strengths

Comprehensive Experimental Evaluation: The paper evaluates multiple VLMs, benchmarks, confidence readouts, and defense strategies. Detailed Analysis: The paper provides detailed analyses of the attack mechanisms through same-answer/cross-answer decomposition and KL-constrained attacks. Practical Implications: The paper demonstrates how confidence manipulation can affect decision-making in confidence-based gating systems.

Critical weaknesses

The paper motivates its study through real-world confidence-based VLM systems, but several evaluated confidence estimators require additional training or model access rather than being directly provided by standard VLMs. Could the authors clarify how representative these confidence estimation settings are of actual VLM deployments?

Decoding strategies: Does the attack remain effective under different decoding strategies, such as temperature or top-p sampling, rather than greedy decoding?

Questions for authors

I wonder how the proposed attack performs under multiple-choice settings with varying numbers of answer choices. Since the paper analyzes same-answer and cross-answer pairs, a controlled evaluation of answer-space size could provide additional insights into the proposed mechanisms.

6 comments
AuthorsOpenReview ↗

Author Response to Reviewer meMC

We thank Reviewer meMC for the careful review and for recognizing the comprehensive evaluation, the decomposition and KL-constrained analyses, and the practical implications for confidence-based gating.

How to read our response. Each row of the table below gives one of your concerns (meMC.1 to meMC.3), quoted from your review, with a short, direct answer beside it. Detailed replies follow in this thread in the same order, titled meMC.1, in detail to meMC.3, in detail. They provide a more detailed response and point to the corresponding changes in the uploaded revised manuscript, where changes are marked in blue.

Your concernOur response
meMC.1 “Several evaluated confidence estimators require additional training or model access rather than being directly provided by standard VLMs”; “Could the authors clarify how representative these confidence estimation settings are of actual VLM deployments?”We found that the vulnerability is not limited to trained probes or estimators that need hidden states: token probability and our newly added sampled-agreement estimator need neither, and both invert. Nor does the attacker need access to model internals: we added score-only attacks through a commercial API, where GPT-5.4-nano’s AUROC falls from 0.6930.693 to 0.1560.156 with its answers unchanged.
meMC.2 “Does the attack remain effective under different decoding strategies, such as temperature or top-p sampling, rather than greedy decoding?”Yes: an attack built for greedy decoding also degrades confidence on freshly sampled answers, without being optimized for sampling. We added an evaluation under temperature and top-p sampling, where SAPLMA AUROC falls from 0.8390.839 to 0.5420.542 on LLaVA and from 0.7270.727 to 0.1770.177 on Gemma.
meMC.3 “How [does] the proposed attack perform under multiple-choice settings with varying numbers of answer choices?”; “Since the paper analyzes same-answer and cross-answer pairs, a controlled evaluation of answer-space size could provide additional insights into the proposed mechanisms.”In the multiple-choice evaluation we added at your suggestion, confidence inverts with two, three, and four answer options, with and without correctness labels. Our decomposition shows that both same-answer and cross-answer discrimination collapse, so changing the size of the answer space does not protect confidence.

Summary. You asked (1) how representative the evaluated confidence estimators are of real deployments, (2) whether the attack holds under sampled decoding, and (3) how it behaves in multiple-choice settings with different numbers of options. We addressed (1) by showing the vulnerability for estimators that need no training or hidden states, and through score-only access to a commercial API; (2) with a new evaluation under temperature and top-p sampling; and (3) with the multiple-choice experiment you suggested, including the decomposition at each option count.

We thank you again for the constructive feedback and invite you to read our detailed replies to each of your three concerns below. We believe they address your concerns and hope you will consider raising your score.

meMC.1, in detail “Several evaluated confidence estimators require additional training or model access rather than being directly provided by standard VLMs”; “Could the authors clarify how representative these confidence estimation settings are of actual VLM deployments?”

Thank you for asking us to clarify this distinction. The vulnerability is not limited to trained confidence probes. We evaluate confidence obtained from token probabilities, repeated model outputs, and learned readouts, covering systems with different access requirements; Section 3 now states which readouts require probe training or hidden-state access.

We first examine confidence estimators that require no probe training:

Confidence estimatorRequirements for computing confidenceFinding
Token probabilityAnswer-token probabilities; no probe trainingBelow chance in all twelve core cells under both correctness-aware and label-free attacks (Appendices D.1–D.2, Tables 21–22)
Sampled agreement (self-consistency)Repeated generated answers; no hidden states or probe trainingLLaVA/GQA: AUROC 0.804→0.0420.804 \rightarrow 0.042; LLaVA/POPE: 0.668→0.0120.668 \rightarrow 0.012 under the correctness-aware attack (Appendix D.8, Table 29)

The sampled-agreement results above establish inversion of informative clean confidence on LLaVA; Gemma’s clean sampled agreement does not reliably distinguish correctness (Appendix D.8, Table 29). These requirements concern computing confidence; the attacks in these experiments use gradient access.

Score-only interfaces are vulnerable without exposing model internals to the attacker. We also test two open-weight models through restricted score-only interfaces, alongside two models accessed through commercial APIs (Section 4; Appendix I.2). The attacker uses returned token-probability scores and responses, without weights, hidden states, or gradients. Correctness-aware SPSA gives the following, taken from Table 51:

ModelBenchmarkClean AUROCScore-only attacked AUROC
LLaVA-OV-7BGQA0.7910.7910.5620.562
InternVL3.5-2BPOPE0.8090.8090.4390.439
GPT-5.4-nanoPOPE0.6930.6930.1560.156
GPT-5.4-miniPOPE0.8580.8580.5610.561

Each model uses a separately constructed balanced 100100-item cohort. Severity varies across settings, but these results demonstrate confidence manipulation through restricted interfaces, including inversion on GPT-5.4-nano through a commercial API.

Trained readouts and independent verifiers cover additional deployment designs. Open-weight deployments can use trained hidden-state probes such as P(IK) and SAPLMA, or perturbation-based estimators such as CCPS (Section 3; Appendix A.1). These require internal access and additional training or computation. Alternatively, a separate verifier can judge an answer without accessing the answering model’s hidden states. We test this arrangement too: verifiers resist transferred attacks better, but lose discrimination when directly targeted (Section 6, Table 2; Appendix H, Table 47).

We do not claim to measure how frequently these estimators are used in production. Rather, we demonstrate the vulnerability across different access settings, including confidence obtained from repeated outputs and probabilities exposed through local score-only interfaces and commercial APIs.

meMC.2, in detail “Does the attack remain effective under different decoding strategies, such as temperature or top-p sampling, rather than greedy decoding?”

Thank you for this question. An attack constructed to preserve the greedy answer also degrades confidence discrimination under sampled decoding. We test our correctness-aware SAPLMA attack with an additional 0.20.2-nat next-token KL constraint (Appendix E). The attack preserves the complete greedy answer but is not optimized for sampled generation.

We evaluate confidence on newly sampled answers. On GQA, we generate 2020 answers per clean and attacked image, using 9999 LLaVA and 9595 Gemma items (Appendix E.1). Each sampled answer is scored against its own correctness. With top-p=0.9p=0.9, we obtain the following, taken from Table 35:

ModelTemperatureClean SAPLMA AUROCAttacked AUROCPaired change, 95% interval
LLaVA-OV-7B0.70.70.8390.8390.5420.542−0.297-0.297 [−0.393,−0.203][-0.393,-0.203]
Gemma-3-12B0.70.70.7270.7270.1770.177−0.550-0.550 [−0.641,−0.459][-0.641,-0.459]
Gemma-3-12B1.01.00.7220.7220.1910.191−0.530-0.530 [−0.621,−0.438][-0.621,-0.438]

All three paired changes exclude zero. LLaVA loses substantial discrimination, while Gemma inverts at both temperatures. Token-probability AUROC also decreases on LLaVA, from 0.7100.710 to 0.5620.562 (Table 35).

The effect persists without a large change in how often the committed answer is sampled. At temperature 0.70.7, its frequency changes from 81.7% to 84.8% on LLaVA and from 86.6% to 83.7% on Gemma. Sampled-answer correctness changes from 58.1% to 52.5% on LLaVA and from 52.9% to 52.4% on Gemma (Table 35). The preservation constraint applies to the greedy answer, not every sampled response.

Reviewer_fJBjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors study whether the confidence a VLM attaches to its answer can be manipulated without changing the answer itself. This matters for systems that use confidence to decide when to trust the model and when to defer to a human. Prior work has shown similar attacks on image classifiers and text-only LLMs. This paper extends the question to open-ended VLM generation, where the full answer must stay fixed.

The authors perturb the input image with white-box PGD while keeping the generated answer byte-identical, lowering confidence on correct answers and raising it on wrong ones. They show that nearly all confidence signals they test, from token probability to trained hidden-state probes, can be pushed below chance, even without knowing which answers are correct. Adversarially trained probes, a separate verifier, and a robust vision encoder all fail once the attacker can reach them. The paper does not propose a defense, and leaves open whether a confidence signal can be both robust and informative.

Critical strengths

The paper studies a well-defined problem with thorough and carefully controlled experiments. Specifically, they provide a completed empirical study of confidence manipulation (while preserving output answers) across various VLMs, benchmarks, and different confidence readouts. Beyond demonstrating the attack's effectiveness, the authors systematically investigate its robustness under different attack assumptions and constraints, including label-free attacks and constraints on the next-token distributions. They also evaluate several potential defenses and examine the downstream impact on confidence-based gating. Together, these experiments provide a well-supported characterization of the vulnerability, its scope, and its practical consequences under the considered threat model.

Critical weaknesses
  1. Insufficient justification of the broader significance: While the paper clearly defines the problem and provides extensive experiments demonstrating the vulnerability of VLM confidence estimates, I am not fully convinced of the broader significance of this investigation. The authors show that confidence manipulation can compromise confidence-based gating, but it remains unclear how relevant the assumed white-box attack setting is to practical VLM deployments. More importantly, beyond demonstrating the vulnerability, the paper provides limited insights into what fundamentally makes confidence estimates vulnerable or how these findings could inform the development of more reliable confidence estimation methods. I would like to see a stronger justification of why this particular problem deserves attention and what generalizable understanding emerges from studying it.

  2. Limited understanding of confidence vulnerability in long-answer generation: While the main experiments demonstrate nearly universal confidence inversion on short-answer VQA benchmarks, the long-answer experiments reveal a more mixed picture. In particular, some hidden-state confidence estimators retain meaningful discrimination under attack on InternVL, whereas they fail on Molmo and Gemma. This difference is especially relevant because preserving an entire generated sequence, rather than a single classification label, is a key distinction from prior work. Although the authors investigate several possible explanations, the underlying reasons for this model-dependent behavior remain unclear. Moreover, the long-answer experiments use a different dataset, prompting strategy, and probe training setup, making it difficult to isolate the role of generation length. A more controlled investigation of these differences would strengthen the paper's understanding of when and why this vulnerability arises in generative VLMs.

Questions for authors

I would be interested to hear the authors' perspective on what insights from this analysis could help guide the development of more robust confidence estimators. For example, the paper identifies a robustness–informativeness trade-off in the tested probes and shows that a robust vision encoder can partially mitigate the attack. Do these observations suggest any broader design principles for improving confidence robustness while preserving the ability to distinguish correct from incorrect answers?

I would also be interested in understanding what underlying factors determine the “reachability” of a confidence signal. Are there particular properties of the model or its representations that might explain why some confidence estimators are more robust than others? It would be helpful to understand whether the reachability analysis offers insights into predicting or improving robustness, beyond characterizing the vulnerability observed under attack.

9 comments
AuthorsOpenReview ↗

Author Response to Reviewer fJBj

We thank Reviewer fJBj for the careful review and for recognizing the thorough, controlled experiments and the well-supported characterization of the vulnerability.

How to read our response. Each row of the table below gives one of your concerns (fJBj.1 to fJBj.4), quoted from your review, with a short, direct answer beside it. Detailed replies follow in this thread in the same order, titled fJBj.1, in detail to fJBj.4, in detail. They provide a more detailed response and point to the corresponding changes in the uploaded revised manuscript, where changes are marked in blue.

Your concernOur response
fJBj.1 “I am not fully convinced of the broader significance of this investigation”; “it remains unclear how relevant the assumed white-box attack setting is to practical VLM deployments.”We show that an attacker can reverse a confidence gate’s decisions without changing any answer, so the attacked gate accepts less accurate answers than using no gate at all. This does not require white-box access: we added score-only attacks through a commercial API, where GPT-5.4-nano’s AUROC falls from 0.6930.693 to 0.1560.156 with its answers unchanged.
fJBj.2 “the long-answer experiments use a different dataset, prompting strategy, and probe training setup, making it difficult to isolate the role of generation length”; “A more controlled investigation of these differences would strengthen the paper's understanding of when and why this vulnerability arises in generative VLMs.”We added the controlled comparison you asked for: the same questions and the same probes, answered once in short form and once with an explanation, with the entire response held exact. We found that explanations make confidence harder to manipulate but do not protect it: both tested readouts invert on all four models, including InternVL.
fJBj.3 “what underlying factors determine the ‘reachability’ of a confidence signal”; “whether the reachability analysis offers insights into predicting or improving robustness, beyond characterizing the vulnerability observed under attack.”Our experiments show that two factors determine reachability: the vision encoder, and the directions of the representation that the readout uses. Reachability can also be anticipated before attacking, since our new clean-image estimate, which accounts for which image changes keep the answer, ranks achieved movement better than plain sensitivity in six of eight model–probe groups.
fJBj.4 “What insights from this analysis could help guide the development of more robust confidence estimators”; “Do these observations suggest any broader design principles for improving confidence robustness while preserving the ability to distinguish correct from incorrect answers?”Our results suggest one design principle: adapt the model’s representation together with the readout, instead of hardening the readout alone. On frozen features, probes gain resistance only by losing clean discrimination, whereas our new joint adaptation of the model and probe raises attacked AUROC from 0.0650.065 to 0.4650.465–0.5090.509 while clean confidence stays informative.

Summary. You asked (1) why this matters beyond white-box access, (2) for a controlled study of longer responses, (3) what determines reachability, and (4) what the analysis implies for building better estimators. We addressed (1) with score-only attacks, including inversion through a commercial API; (2) with a controlled comparison of short and explanation-bearing responses on the same questions; (3) by showing that the representation and the readout both matter, and with a new estimate of which items are movable; and (4) with joint adaptation of the model and probe, which improves resistance.

We thank you again for the constructive feedback and invite you to read our detailed replies to each of your four concerns below. We believe they address your concerns and hope you will consider raising your score.

fJBj.1, in detail “I am not fully convinced of the broader significance of this investigation”; “it remains unclear how relevant the assumed white-box attack setting is to practical VLM deployments.”

Thank you for raising this. The practical concern is that confidence gates allocate both trust and human attention. We show that an attacker can reverse that allocation, sending errors into automated acceptance and correct answers into unnecessary review. Our new experiments show that this problem extends beyond the assumed white-box attack.

We demonstrate confidence manipulation without gradient access. We added evaluations of two local models through score-only interfaces and two commercial API models, using a separately constructed balanced 100100-item cohort for each model and 600600 search queries per item and direction (Section 4; Appendix I.2). The attacker observes confidence scores and responses, without gradients or hidden states; correctness labels select the harmful endpoint.

ModelBenchmarkClean AUROCRandom searchSPSA
LLaVA-OV-7BGQA0.7910.7910.6110.6110.5620.562
InternVL3.5-2BPOPE0.8090.8090.5120.5120.4390.439
GPT-5.4-nanoPOPE0.6930.6930.1940.1940.1560.156
GPT-5.4-miniPOPE0.8580.8580.4980.4980.5610.561

Our new results (Appendix I.2, Table 51) show that confidence can be corrupted without changing the answer even when the attacker cannot access the model’s gradients. White-box experiments test how far this manipulation can go; our score-only experiments show that the phenomenon persists without that access. On GPT-5.4-nano, we drive discrimination far below chance through a commercial API. Applying the same correctness-aware selection to repeated unperturbed requests leaves AUROC at 0.6340.634 (the no-perturbation control in Table 51), so ordinary API response variation does not explain the inversion.

We show how this fools the automation decision. In our local deployment experiments, a label-free attack makes the gate accept up to 79.0% of previously rejected wrong answers (Appendix F.1, Table 36). At each model’s natural error rate, the answers accepted by the attacked gate are less accurate than using no gate and automatically accepting every model answer, in all twelve model–benchmark cells (Table 36, middle block). We further demonstrate that an image supplier can first induce an error and then raise its confidence enough to fool the gate: up to 84% of initially correct items become accepted wrong answers within one perturbation budget (Section 4; Appendix F.4, Table 39). We therefore show more than movement in a confidence score: we fool the check intended to catch errors and pass wrong answers into automation. We address your related questions about why confidence is vulnerable and how these findings guide better estimators in fJBj.3 and fJBj.4.

We also added a full-token-verified gate evaluation (Appendix F.2, Table 37), and the combined attack's stage-2 results now use the strict token-preserving evaluation (Table 39).

fJBj.2, in detail “the long-answer experiments use a different dataset, prompting strategy, and probe training setup, making it difficult to isolate the role of generation length”; “A more controlled investigation of these differences would strengthen the paper's understanding of when and why this vulnerability arises in generative VLMs.”

Thank you for suggesting a more controlled investigation. We added a comparison using the same benchmark, questions, and probe bank for short and explanation-bearing responses (Section 5; Appendix C.5). This directly tests whether our finding extends beyond short answers without changing the dataset or retraining the confidence probes.

We compare both formats on the same questions. In our new experiment, all four models answer the same 200200 POPE questions, one per source image. Median response lengths increase from 11–22 tokens for short answers to 1414–2323 tokens when an explanation accompanies the answer. We retain the existing probes and directly attack answer-span token probability and P(IK) within the same 8/2558/255 image budget. Every accepted image must reproduce the complete committed response, including the explanation and termination. The intervention changes response format, including its associated length.

We show how confidence changes in each format. Each entry below gives clean →\rightarrow attacked AUROC, taken from Table 14 in Appendix C.5. Both formats receive 6060 steps and five restarts, with correctness-aware direction selection.

ModelReadoutShort responseAnswer with explanation
InternVL3.5-2BTokProb0.727→0.0000.727 \rightarrow 0.0000.711→0.1550.711 \rightarrow 0.155
InternVL3.5-2BP(IK)0.743→0.0000.743 \rightarrow 0.0000.603→0.1570.603 \rightarrow 0.157
Molmo2-4BTokProb0.642→0.0090.642 \rightarrow 0.0090.715→0.3460.715 \rightarrow 0.346
Molmo2-4BP(IK)0.714→0.0000.714 \rightarrow 0.0000.718→0.2780.718 \rightarrow 0.278
LLaVA-OV-7BTokProb0.689→0.0000.689 \rightarrow 0.0000.699→0.1130.699 \rightarrow 0.113
LLaVA-OV-7BP(IK)0.717→0.0000.717 \rightarrow 0.0000.771→0.1270.771 \rightarrow 0.127
Gemma-3-12BTokProb0.560→0.0000.560 \rightarrow 0.0000.748→0.3910.748 \rightarrow 0.391
Gemma-3-12BP(IK)0.675→0.0110.675 \rightarrow 0.0110.717→0.2460.717 \rightarrow 0.246

Explanation-bearing responses suffer less degradation in all eight paired comparisons at 3030 steps and two restarts, and seven of eight at 6060 steps and five restarts, with intervals excluding zero (Appendix C.5, Table 15). Nevertheless, all eight stronger-search attacked intervals remain below chance (Table 14). In particular, InternVL also inverts, reaching 0.1550.155 for token probability and 0.1570.157 for P(IK). Its probes’ resistance in the longer-response experiment therefore does not extend to this controlled setting: vulnerability depends on the response setting, not simply the model identity.

We extend the investigation to substantially longer responses. On the separate A-OKVQA benchmark, responses have median lengths of 7777–9797 tokens (Section 5; Appendix C.6). Final-answer preservation permits P(IK) and SAPLMA inversion on Molmo and Gemma, reaching 0.3800.380–0.4330.433. Requiring the entire response to remain exact limits the tested attack, leaving these probes at 0.5950.595–0.6600.660 across the four models. Token probability nevertheless inverts under whole-response preservation on three models (Appendix C.6, Table 17).

Confidence inversion extends beyond short answers in a controlled comparison. We believe these experiments directly answer your concern: we establish explanation-bearing inversion with shared questions and unchanged probes, and show how preserving the complete response restricts the manipulation achieved on substantially longer responses.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Overview of Author Response and Revision

Dear Area Chair,

We thank you and the three reviewers for the time and consideration given to our submission. This note summarizes the review discussion and what changed in the uploaded revised manuscript, where changes are marked in blue.

Summary of the reviews. All three reviewers found the experimental evaluation thorough and recognized its practical consequences for confidence-based gating. The concerns fell into four groups: (1) whether the problem matters beyond a white-box setting, (2) what explains why confidence can be manipulated and how to build better estimators, (3) coverage of models, confidence estimators, and defenses, and (4) whether the finding holds under other response and decoding settings.

What we did. We ran new experiments for each group and revised the paper accordingly. The table lists them with the reviewers whose concerns each one answers.

New experimentMain resultAnswers
Score-only attacks, including a commercial APIGPT-5.4-nano’s AUROC falls from 0.6930.693 to 0.1560.156 without gradients or weightsfJBj, meMC, TwMU
Controlled short versus explanation-bearing responsesConfidence inverts on all four models with the full response held exactfJBj, TwMU
Preservation-aware estimate of which items are movableRanks achieved movement better than plain sensitivity in six of eight groupsfJBj, TwMU
Joint adaptation of the model and probeAttacked AUROC rises from 0.0650.065 to 0.4650.465–0.5090.509 with clean confidence still informativefJBj, TwMU
Qwen2.5-VL at 3B, 7B, and 32BConfidence inverts at every sizeTwMU
Self-consistency and verbalized confidenceBoth fall below chance under direct attackmeMC, TwMU
Combined readoutThe fitted combination inverts when targeted directlyTwMU
Second robust vision encoder (TeCoA)Same pattern as FARE, with inversion at the headline budgetTwMU
Trained verifiers and their combinationsA trained verifier inverts under direct attack, and combining verifiers does not prevent itTwMU
Adaptive attacks on JPEG and randomized smoothingBoth defenses fail once the attacker accounts for themTwMU
Sampled decoding (temperature and top-p)Confidence discrimination degrades on sampled answersmeMC
Multiple choice with two, three, and four optionsConfidence inverts at every option countmeMC

We also revised the text where reviewers asked: the paper now distinguishes readouts from method families, limits its conclusions to what was tested, describes the verifiers accurately, and states the threat model in the abstract and introduction.

Outcome. The new experiments confirm the paper’s central claim and extend it: confidence can be manipulated while the answer stays fixed, across model families and sizes, across estimator types, under the defenses reviewers proposed, and without white-box access. They also add what the reviewers asked for beyond the attack itself, namely a tested account of what makes confidence movable and a first defense that improves resistance.

Where to find our responses. Each reviewer thread opens with a table that briefly answers each concern, followed by a detailed reply per concern with results and pointers to the revised paper. Concerns are labelled fJBj.1 to fJBj.4, meMC.1 to meMC.3, and TwMU.1 to TwMU.10.

We believe the revision addresses the concerns the reviewers raised. We remain available for any further questions or comments during the discussion period.

Thank you.

AuthorsOpenReview ↗

Update on the Reviewer Discussion and Revision

Dear Area Chair,

A brief update on the discussion since our overview above.

Reviewer responses:

  • Reviewer meMC wrote that "the authors have adequately addressed my concerns" and maintains a positive rating.
  • Reviewer TwMU replied point by point, marking concerns as resolved and welcoming the added experiments. The reviewer described the response as an "overall good rebuttal effort" and wrote that the "empirical side is now more complete".
  • Reviewer fJBj wrote that the additions "make the empirical picture much clearer and address several of my original questions" and that "the empirical side of the paper is now much stronger". Our reply to the reviewer's follow-up is in that thread.

What changed since the overview. Following Reviewer TwMU's follow-up, we added one comparison the reviewer suggested (Section 5; Appendix I.5) and improved the clarity of the writing throughout. The results, claims, and numbering are unchanged.

We remain available for any further questions.

Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.