The Doubt Is Not Gone, Only Moved: Anti-Calibrated Confidence and a Free Denoising-Trajectory Hallucination Detector for Diffusion VLMs (SEISMO)

A model's confidence is usually treated as a property of its weights. In a masked-diffusion vision–language model (VLM), we show it is a property of how the model decodes. At each denoising step the sampler fills in the answer positions it is most sure about and re-masks the rest. The final distribution is therefore sharpened by the sampler itself, so it looks equally certain whether the answer is right or wrong. This creates a reliability trap. On LLaDA-V, output confidence—the score almost every abstention method uses—cannot find the model's own mistakes: it is at chance on POPE (0.50 AUROC) and below chance on GQA (0.37–0.39). A confident answer is no safer than a random one. So where did the doubt go? It is not gone, only moved. The step-by-step denoising trajectory still records which answer positions were contested, and for how long. Simply averaging the answer position's per-step entropy detects wrong answers at 0.65–0.79 AUROC across four diffusion VLMs. It costs nothing, because it is read off the single decode the model already runs. A simple regularity says where it helps: the further a model's trajectory ranking drifts from its endpoint ranking, the larger the gain. SEISMO is our answer to that question—to our knowledge the first reliability framework built for diffusion VLMs. A calibration head cuts expected calibration error by $2.7\times$. A Learn-then-Test gate then adds the first distribution-free hallucination-risk guarantee for a diffusion VLM: it answers the most reliable half of POPE at 95% accuracy, where trusting confidence gives 79%. The signal diagnoses but cannot repair: the visual evidence is lost before the decoder runs. It does route, though: sending flagged questions to an autoregressive fallback matches full replacement at a fifth of the cost ($0.830\to 0.868$), while routing by confidence makes things worse. The lesson is general: a reliability method can silently invert, not merely weaken, when carried to a new kind of generation—and the check that catches this is free. All per-example records are released.