Vision-Language Model Confidence Is Not a Property of the Answer

Vision-language models are increasingly deployed behind a confidence gate: the system reads how confident the model is in its answer and defers when that confidence is low. This makes the confidence signal itself worth attacking. We show that a white-box adversary who perturbs only the input image, within an $L_{\infty }$ budget of $8/255$ and under the constraint that the model's answer stay byte-identical, can invert the confidence ranking, lowering it on correct answers and raising it on wrong ones until the signal points the wrong way. The answer never changes, and most of the inversion persists even when the whole next-token distribution is held near the clean one, so the answer does not determine the confidence attached to it. Across four vision-language models and three visual question-answering benchmarks, the attack drives the model-internal readouts we test below chance in 83 of 84 readout-by-cell profiles under an adversary that knows which answers are correct, and six of the seven readouts fall below chance in every cell under an adversary that does not. Training a probe on the model's frozen hidden states does not fix this: the robustness such a probe gains against the attack is paid for with the information that made it useful. Nor does reading confidence from a separate verifier model, which holds only until the attacker can reach it. Nor does an adversarially trained vision encoder, which at its own training budget holds against the attacker without correctness labels but not against the one with them. How far an answer-preserving adversary can reach a signal governs where it survives, and whether a robust and informative readout can be built remains open. For deployment, a gate under this attack admits up to $89.8\%$ of the wrong answers it would otherwise catch, $79.0\%$ when the attacker has no correctness labels, and, corrected for how often the model is wrong, the answers it accepts are less accurate than with no gate at all.