acceptodds
Under review as a conference paper at ICLR 2027

The Doubt Is Not Gone, Only Moved: Anti-Calibrated Confidence and a Free Denoising-Trajectory Hallucination Detector for Diffusion VLMs (SEISMO)

Abstract

A model's confidence is usually treated as a property of its weights. In a masked-diffusion vision–language model (VLM), we show it is a property of how the model decodes. At each denoising step the sampler fills in the answer positions it is most sure about and re-masks the rest. The final distribution is therefore sharpened by the sampler itself, so it looks equally certain whether the answer is right or wrong. This creates a reliability trap. On LLaDA-V, output confidence—the score almost every abstention method uses—cannot find the model's own mistakes: it is at chance on POPE (0.50 AUROC) and below chance on GQA (0.37–0.39). A confident answer is no safer than a random one. So where did the doubt go? It is not gone, only moved. The step-by-step denoising trajectory still records which answer positions were contested, and for how long. Simply averaging the answer position's per-step entropy detects wrong answers at 0.65–0.79 AUROC across four diffusion VLMs. It costs nothing, because it is read off the single decode the model already runs. A simple regularity says where it helps: the further a model's trajectory ranking drifts from its endpoint ranking, the larger the gain. SEISMO is our answer to that question—to our knowledge the first reliability framework built for diffusion VLMs. A calibration head cuts expected calibration error by . A Learn-then-Test gate then adds the first distribution-free hallucination-risk guarantee for a diffusion VLM: it answers the most reliable half of POPE at 95% accuracy, where trusting confidence gives 79%. The signal diagnoses but cannot repair: the visual evidence is lost before the decoder runs. It does route, though: sending flagged questions to an autoregressive fallback matches full replacement at a fifth of the cost (), while routing by confidence makes things worse. The lesson is general: a reliability method can silently invert, not merely weaken, when carried to a new kind of generation—and the check that catches this is free. All per-example records are released.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
diffusion language models, vision-language models, hallucination detection, uncertainty quantification, calibration, selective prediction, distribution-free risk control, reliability

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kLn1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies reliability in diffusion vision-language models (VLMs). The authors show that on dynamically decoded LLaDA-family diffusion VLMs, final output confidence fails to detect the model’s own errors: AUROC is only 0.50 on POPE (chance) and 0.37–0.39 on GQA (below chance). They attribute this to the sampler committing high-confidence positions during denoising, which sharpens the final distribution and erases the model’s internal doubt. As an alternative, they read the per-step entropy at the answer position along the denoising trajectory and average it (mean step entropy), yielding a training-free, single-pass error-detection signal. This signal reaches 0.65–0.79 AUROC across four diffusion VLMs. They further build SEISMO: a logistic calibration head reduces ECE by 2.7×, and a Learn-then-Test gate provides a distribution-free hallucination-risk guarantee. Experiments cover POPE, GQA, MME, HallusionBench, AMBER, and medical VQA, with boundary analyses for open-ended generation, the medical domain, and non-LLaDA backbones. They also show the signal can diagnose but not repair errors, though it can route suspicious questions to an AR fallback.

Critical strengths
  1. An important and counterintuitive finding: output confidence in diffusion VLMs can be anti-calibrated. This directly challenges reliability methods transferred from AR models. It has broad implications for the multimodal reliability and uncertainty-quantification communities, because it shows that a reliability method can silently invert under a new generation paradigm rather than merely degrade.

  2. A training-free, single-pass trajectory signal: mean step entropy is read directly from the model’s existing decode, adds no compute, and is stable across multiple models and benchmarks. Combined with a calibration head and Learn-then-Test risk control, it provides a full pipeline from detection to guarantees. This is the first distribution-free hallucination-risk guarantee for diffusion VLMs and has practical deployment potential.

Critical weaknesses

Generality is limited to LLaDA-family models and short-answer QA: the advantage appears only on dynamically decoded LLaDA-initialized models; forcing the Dream backbone into dynamic decoding did not transfer the signal. The signal also fails in open-ended generation. This makes the method more of a diagnostic tool for a specific model family than a general solution for diffusion VLMs. Given the claim of a reliability framework for diffusion VLMs, the scope is narrow.

Questions for authors
  1. Have you tried replacing “mean step entropy” with “mean step max probability”? Both can separate correct from incorrect answers in early steps, but entropy is stronger. Can you provide this ablation to show which part of the distribution shape drives entropy’s advantage?

  2. In open-ended generation, mean step entropy is highly correlated with commit step (rank corr. 0.84). After residualizing on commit step, do any other trajectory statistics retain signal? Have you tried grouping mentions by commit stage?

  3. For non-LLaDA backbones such as Dream, the advantage did not transfer even under forced dynamic decoding. Have you analyzed how Dream’s early-step entropy distribution differs fundamentally from LLaDA’s? Does this imply the trajectory signal depends on a specific representational structure?

  4. SEISMO-Guard requires a labeled calibration set. In deployment scenarios where same-distribution labeled data are unavailable, have you considered unsupervised or weakly supervised ways to estimate the calibration threshold? The conservative transfer property fails under upward drift; can the sequential e-process monitor provide early warning rather than only post-hoc alarms?

5 comments
AuthorsOpenReview ↗

Response to Reviewer kLn1 (1/2)

Thank you for the positive review and the sharp questions. Each one gave us new results. Together they make SEISMO broader than the paper first claimed.

Main concern. "Generality is limited to LLaDA-family models and short-answer QA: the advantage appears only on dynamically decoded LLaDA-initialized models; forcing the Dream backbone into dynamic decoding did not transfer the signal. The signal also fails in open-ended generation. This makes the method more of a diagnostic tool for a specific model family than a general solution for diffusion VLMs. Given the claim of a reliability framework for diffusion VLMs, the scope is narrow."

The paper already maps where the step signal helps and where it does not (Section 5.3 on page 6 and the Limitations on page 9). We have now gone further. SEISMO is not limited to LLaDA models. We tested three model families. SEISMO beats the model's own confidence in 5 of 9 tests. It ties in the other 4. It never loses.

Model (family)POPE confidence → SEISMOGQA confidence → SEISMO
Nemotron Diffusion VLM (Nemotron)0.51 → 0.820.64 → 0.80
LaViDa-LLaDA (LLaDA)0.79 → 0.790.60 → 0.79
Dream-VL (Dream)0.82 → 0.800.66 → 0.74
Dimple (Dream)0.78 → 0.800.78 → 0.85
LaViDa-Dream (Dream)0.82 → 0.82not tested

This turns a limit into one of SEISMO's strongest results. One simple rule holds across three families. If the answer is read after the model has written its own text then confidence fails and SEISMO recovers the signal. This happens with long answers (LLaDA-V and Nemotron). It also happens on hard questions where the answer is placed late (every GQA test). We read every Dream model through its own built in hook. Our earlier LaViDa-Dream capture read the answer one position too late. These new numbers replace those results and the forced Dream test in Appendix B.10. The POPE conclusion does not change.

Q1. "Have you tried replacing “mean step entropy” with “mean step max probability”? Both can separate correct from incorrect answers in early steps, but entropy is stronger. Can you provide this ablation to show which part of the distribution shape drives entropy’s advantage?"

Section 3.2 (page 4) already uses both final max probability and final entropy as baselines. Here is the full ablation over all steps that you asked for.

AUROCLLaDA-V POPELLaDA-V GQA
SEISMO score (entropy over all steps)0.760.72
Average top probability over all steps0.760.70
Final confidence (last step only)0.470.37

What matters is reading the whole decoding path. That is the SEISMO idea. Both path based scores beat final confidence by about 0.3. On yes or no questions the two scores are almost the same (rank correlation 0.97). On open GQA questions entropy is a bit better. It also sees how the chance is spread over other possible answers. That part alone scores 0.75. This is why SEISMO uses entropy.

AuthorsOpenReview ↗

Response to Reviewer kLn1 (2/2)

Q2. "In open-ended generation, mean step entropy is highly correlated with commit step (rank corr. 0.84). After residualizing on commit step, do any other trajectory statistics retain signal? Have you tried grouping mentions by commit stage?"

The paper already measures this link and already removes the commit step from our score (Section 6 on page 9 and Appendix G on page 34). Now we also did this for the other step signals and grouped the mentions by stage as you asked. After removing the commit step the signal read after the answer settles still finds wrong objects at 0.61 to 0.62. This is the same level as confidence (0.60 to 0.61). So the path carries real content and not only decoding order.

Commit stageEarlyMiddleLate
Path signal after the answer settles0.570.560.68
SEISMO score (all steps)0.480.530.65

The signal is strongest for objects placed late. SEISMO also maps where path based detection helps most and explains why. In short answer questions the answer is contested and SEISMO leads by a lot. In long free captions the score partly becomes a decoding order clock. So there it matches confidence instead of beating it (Section 6).

Q3. "For non-LLaDA backbones such as Dream, the advantage did not transfer even under forced dynamic decoding. Have you analyzed how Dream’s early-step entropy distribution differs fundamentally from LLaDA’s? Does this imply the trajectory signal depends on a specific representational structure?"

The paper already points to the answer in Section 6 (page 8). The gain appears where decoding squeezes the early doubt out of the final answer. Here we show this directly with the shape of the doubt curve. The table shows the median answer entropy on POPE.

ModelDoubt at step 1Doubt at the last stepHow much it shrinks
LLaDA-V2.110.003about 830 times
LaViDa-Dream0.110.101.1 times
Dream-VL0.700.302.4 times
Dimple0.270.211.3 times

LLaDA-V starts very unsure. Decoding then squeezes the doubt out of the final answer. Only the path keeps it. Dream based models start with far less doubt and lose much less of it by the last step. So their final confidence still holds the doubt and works well (0.78 to 0.82). It is not about a special inner structure. When Dream models answer late (GQA) SEISMO wins on them too. This is a direct test of how SEISMO works. The gain shows up exactly where decoding squeezes out early doubt.

Q4. "SEISMO-Guard requires a labeled calibration set. In deployment scenarios where same-distribution labeled data are unavailable, have you considered unsupervised or weakly supervised ways to estimate the calibration threshold? The conservative transfer property fails under upward drift; can the sequential e-process monitor provide early warning rather than only post-hoc alarms?"

  • Setting the threshold. Every error guarantee needs some labels. Only a label can tell us that an answer is wrong. SEISMO needs just one labeled set at the start. Appendix C.5 (page 29) already shows that the same threshold then stays safe on easier data with no new labels.
  • Early warning without labels. Yes. The paper already has a monitor that uses labels (Table 12 on page 29). We have now added a second monitor that needs no labels at all. It simply counts how often SEISMO holds back an answer. If SEISMO starts holding back more answers than it did on POPE then the questions have become harder and the monitor raises an alarm. We set it up on POPE. Then we ran it on the other POPE splits and on four harder benchmarks.
Used onIs the guarantee still safe?How often the new monitor raised an alarmQuestions before the alarm (median)
POPE random and popular and adversarialyes1% or lessnot needed
HallusionBenchno100%36
AMBERno100%58
MMEno100%211
GQAno100%5

The new monitor raised an alarm every time the guarantee broke. It did this without a single label. When nothing changed it gave a false alarm only 2.6% of the time. The labeled monitor caught HallusionBench and AMBER only 6% and 25% of the time. On these two the error is only a little above the limit so labels give a weak signal. The number of held back answers changes much more and the new monitor catches it every time.

With this SEISMO covers every stage of use. It sets a safe threshold at the start. It warns early without labels. It also double checks once labels arrive.

Thank you for the authors’ additional clarifications. The research direction is potentially meaningful, but the manuscript’s writing and argumentative structure remain insufficiently clear. The connections among the motivation, methodology, and main contributions are not always well articulated, which makes the central message difficult to follow.

Reviewer_MkdXOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper looks at the reliability of output confidence in masked-diffusion VLMs (mainly LLaDA-V, plus LaViDa, MMaDA and LLaDA-MedV). It shows that on LLaDA-V the final answer confidence is at or below chance for detecting wrong answers on POPE and GQA, and attributes this to the confidence-based remasking of the sampler, which sharpens the final distribution. Instead, the authors read the denoising trajectory: the mean per-step entropy at the answer position is free to compute and detects errors. On top of it they build SEISMO, with a logistic calibration head, a Learn-then-Test gate for risk control, and routing of flagged questions to an AR model. They also show that the signal does not help to correct the answers (TCR, step voting, DPO).

Critical strengths
  • Scientifically, i believe that the paper has some interersting approach, and the numbers seem to support the claims.
  • The observation that the confidence of diffusion VLMs can be uninformative (or even anti-calibrated), while the trajectory still holds the uncertainty, is interesting and useful as these models get more common. The signal is also free, read from the single decode.
  • Several models and benchmarks are evaluated, and per-example records and code are released.
Critical weaknesses

W1: The full paper reads really a lot like an LLM-based paper. But most of the characteristics of Opus 5.1 can be found here, a lot of technical words; a lot of numbers, including confidence intervals that would not be required, very complex wordings; focus on minor details when bigger picture information are simply skipped.

W2: Many terms are undefined in the paper. n_changes, commit_step, conf_at_commit, temporal_incons, mean_step_entropy, etc

W3: the supplementary should be optional, not mandatory to understand the paper!

W4: Currently, most of the related work is absent.

W5: Same for the method section: It is really short, main scores should be defined in the manuscript; TCR should also be defined in manuscript.

W6: Experimental sections seem to include all the experiments that were done with claude, and not a filtered version of the experiment that actually supports the claim in the paper (also illustrated by the artifact provided in the link), this is also quite clear from the lack of definition of many variables.

W7: The current version of the paper is far from ready to be accepted. It would require a MAJOR rewriting in my opinion.

W8: How is AUROC equivalent to the Mann–Whitney statistic ?

W9: “sequential e-process monitor over realized kept-error”: What does that mean?

W10: Claims are broader than what is shown: the anti-calibration only appears on LLaDA-initialized models with dynamic decoding (as said in the limitations), and the risk control only gives useful coverage on low-error POPE-style questions. Calling it “the first distribution-free hallucination-risk guarantee” is a bit strong, it is a standard Learn-then-Test applied on a new score.

Questions for authors

See weaknesses.

8 comments
AuthorsOpenReview ↗

Response to Reviewer MkdX (1/3)

W1. "The full paper reads really a lot like an LLM-based paper. But most of the characteristics of Opus 5.1 can be found here, a lot of technical words; a lot of numbers, including confidence intervals that would not be required, very complex wordings; focus on minor details when bigger picture information are simply skipped."

W3. "the supplementary should be optional, not mandatory to understand the paper!"

W5. "Same for the method section: It is really short, main scores should be defined in the manuscript; TCR should also be defined in manuscript."

W7. "The current version of the paper is far from ready to be accepted. It would require a MAJOR rewriting in my opinion."

The key parts you ask for are already in the paper. Our main score is defined in Section 3.2 and TCR is introduced in Section 3.3 (both on page 4). Every number in the paper comes from our released scripts and records. Anyone can check them on a normal CPU.

We will move the full TCR rule and Table 3 into Section 3. With this the main text will stand on its own and the appendix will be optional.

W2. "Many terms are undefined in the paper. n_changes, commit_step, conf_at_commit, temporal_incons, mean_step_entropy, etc"

All of these terms are already defined in the paper. Section 3.2 (page 4) explains our main score in words. It also points the reader to Table 3 and Algorithm 1. Table 3 in Appendix A.1 (page 15) defines every symbol and signal name. Algorithm 1 (page 18) gives the exact formula for each signal. TCR is introduced in Section 3.3 (page 4) and its full rule is in Appendix A.7 (page 18).

TermWhere it is defined in the paperMeaning
mean_step_entropy (SEISMO score)Section 3.2 (page 4) and Table 3 (page 15) and Algorithm 1 (page 18)How unsure the model is about the answer on average over all steps
n_changesTable 3 (page 15) and Algorithm 1 (page 18)How many times the model's best guess changes
commit_stepTable 3 (page 15) and Algorithm 1 (page 18)The last step where the best guess was different from the final answer
temporal_inconsTable 3 (page 15) and Algorithm 1 (page 18)The share of steps that disagree with the final answer
conf_at_commitTable 3 (page 15) and Algorithm 1 (page 18)How unsure the model is when the answer settles
final_maxprob and final_entropyTable 3 (page 15)The usual confidence read at the last step
TCRSection 3.3 (page 4) and Table 3 (page 15) and Appendix A.7 (page 18)A decoding rule that places a word only when the guess is stable and the doubt is low
AuthorsOpenReview ↗

Response to Reviewer MkdX (2/3)

W4. "Currently, most of the related work is absent."

Related work is part of the paper. Section 2 (page 4) is a full Related Work section in the main text. Appendix I (pages 37 and 38) gives a longer version. Table 15 (page 38) places SEISMO next to each line of work. In total the paper cites 59 works. The papers closest to ours are all discussed in Section 2 of the main text. These are TraceDet (Chang et al. ICLR 2026) and DynHD (Qian et al. 2026) and HIVE (Zhao et al. 2026) and Vazhentsev et al. (2026).

Line of workWhere it is in the paperWhat these papers doWhat SEISMO adds
Uncertainty for diffusion language models (TraceDet and DynHD and HIVE and Vazhentsev et al. and four more)Section 2 (page 4) and Appendix I (page 37)Read the decoding steps but only for textBrings the step signal to vision language models and shows that their own confidence can fail
Hallucination detection for VLMs (Leng et al. and Wu et al. and Jiang et al. and OPERA and VAUQ)Section 2 (page 4) and Appendix I (pages 37 and 38)Built for AR models and rely on the final confidenceShows this confidence can fail on diffusion VLMs and gives a score that works. A VAUQ style baseline is run in Appendix B.6 (page 23)
Safe answering with error guarantees (Tayebati et al. and Xu et al.)Section 2 (page 4) and Appendix I (page 38)Give guarantees for AR VLMs but take confidence as inputSupplies the score that makes the guarantee useful on diffusion VLMs
Decoding order and fast decoding (Lee et al. and Jazbec et al. and Saini et al. and Wang et al. and VISAGE)Appendix I (pages 37 and 38)Change the unmasking order for speed or qualityDetects errors and gives a guarantee instead
Classic uncertainty and calibration (semantic entropy and deep ensembles and temperature scaling)Table 15 (page 38)Need many runs or look only at the final outputNeeds one normal run and reads every step

We thank Reviewer ELvu for pointing us to VGR (Song et al. ICML 2026) and we will cite it. We will also move Table 15 into Section 2 so the main text shows at a glance where SEISMO stands.

W6. "Experimental sections seem to include all the experiments that were done with claude, and not a filtered version of the experiment that actually supports the claim in the paper (also illustrated by the artifact provided in the link), this is also quite clear from the lack of definition of many variables."

The paper already lists its contributions in order of importance (Section 1 on pages 2 and 3). All variables are already defined (see W2). The main text will be built around five contributions and each gets one table or figure.

  1. The model's own confidence fails on diffusion VLMs. The decoding path keeps the doubt.
  2. A simple rule says when it fails. It fails when the answer is read after the model has written its own text. This holds across three model families.
  3. The SEISMO score needs no training and works across models. Trained detectors flip and fail on new models.
  4. The error guarantee works with the SEISMO score. It does not work with normal confidence.
  5. Detection leads to action. Sending flagged questions to a backup model matches full replacement at one fifth of the cost.
AuthorsOpenReview ↗

Response to Reviewer MkdX (3/3)

W8. "How is AUROC equivalent to the Mann–Whitney statistic ?"

Section 3.2 (page 4) already says that we compute AUROC with the Mann–Whitney rank method. Here is why the two are the same. AUROC is the chance that a random wrong answer gets a higher score than a random correct answer. Ties count as half. The Mann–Whitney statistic U counts exactly these pairs. So AUROC = U / (number of wrong × number of correct). On POPE all three ways give the same number.

Counting pairsU / (n1 × n0)Area under ROCMann–Whitney p
SEISMO0.77810.77810.77815 × 10⁻¹³
Confidence0.50400.50400.50400.92

W9. "“sequential e-process monitor over realized kept-error”: What does that mean?"

The monitor is already described in Section 5.5 (page 7) and Appendix C.5 (page 29). Its results are in Table 12. In plain words it works like this. After launch the true labels of answered questions come in one by one. The monitor makes a running bet that the error rate is above the promised level. If the promise holds the bet loses money on average. So the chance that it ever grows to 10 times its start is at most 10%. This is a known result (Ville's inequality). When the bet reaches 10 times its start we raise an alarm. The check is valid after every new label.

SituationAlarm rateAnswers needed
No drift4.5% (below the 10% limit)none
Error rate rises to 0.35100%47 (median)

W10. "Claims are broader than what is shown: the anti-calibration only appears on LLaDA-initialized models with dynamic decoding (as said in the limitations), and the risk control only gives useful coverage on low-error POPE-style questions. Calling it “the first distribution-free hallucination-risk guarantee” is a bit strong, it is a standard Learn-then-Test applied on a new score."

The paper already states its scope openly in the Limitations section (page 9) and Appendix J (page 38). Section 5.5 (page 7) already says where the guarantee gives the most value.

  • The value is in the score. The guarantee method is standard on purpose. On the same data normal confidence can safely answer only 6% of POPE at α = 0.10. SEISMO answers 51% (1200 questions). As far as we know SEISMO is the first error guarantee for abstention in a diffusion VLM.
  • The finding is not only for LLaDA models. On Nemotron Diffusion VLM (not built on LLaDA) the model's own confidence is at chance on POPE (0.51). SEISMO reaches 0.82. The gain is 0.31 and it is significant. SEISMO never loses in any new test (see our reply to kLn1).
Reviewer_ELvuOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies reliability estimation in masked-diffusion vision–language models. Its central empirical observation is that final-step confidence on LLaDA-V fails to rank incorrect answers (POPE/GQA error-detection AUROC approximately 0.50/0.39), whereas mean entropy at the first answer position across denoising steps gives 0.778/0.723. The authors examine multiple diffusion VLMs and an autoregressive mirror, relate performance to trajectory-endpoint rank decoupling, and introduce SEISMO: a training-free trajectory score, a five-feature logistic calibration head, and a Learn-then-Test selective-answering gate. On POPE the score lowers risk at 50% coverage from 21% to 5%, and routing 20% of queries to an AR fallback improves system accuracy from 0.830 to 0.868. The paper also reports negative in-stack correction results and a loss of advantage on open-ended COCO captioning.

Critical strengths
  1. Strong empirical reliability diagnosis. The gap between trajectory entropy and endpoint confidence is large on LLaDA-V (POPE 0.778 vs. 0.500; GQA 0.723 vs. 0.385), with interval estimates, matched examples, prompt and decoding controls, and some cross-model replication. This is a consequential and practically interpretable finding, rather than a small metric improvement.

  2. Careful accounting of limitations and actionable use. The paper reports that in-stack remasking and voting do not recover correctness, acknowledges failure on open-ended captioning and difficult medical selective-risk settings, and evaluates routing as a distinct system-level intervention. These negative and boundary results materially strengthen the empirical contribution.

Critical weaknesses
  1. The finite-sample risk guarantee needs a stronger validity argument. Appendix A.8 states that the candidate keep-fractions are 10%, 15%, 100% of the calibration sample, ordered by the same sample's reliability scores, and then uses Clopper–Pearson bounds in fixed-sequence testing. This makes the numerical score cutoffs calibration-sample-dependent. Standard fixed-sequence LTT arguments require valid p-values for each candidate policy. The exact binomial argument for a fixed threshold does not automatically apply to a cutoff obtained as an order statistic from those same observations. The paper should prove validity for this construction or use thresholds fixed independently of the risk-calibration sample and rerun the reported certification experiments. The 200 repeated splits are an empirical check, not a substitute for the claimed distribution-free result.

  2. The novelty and primary confidence comparison need clarification, Song et al (ICML 2026), already exploits confidence trajectories to detect visual hallucination in multimodal diffusion models, but is absent from the paper's positioning. SEISMO may still contribute a distinct score, AR comparison, and risk-controlled abstention, but its first/missing-intersection claims should be qualified and a closer baseline considered. Separately, the diffusion final-step logits may be computed after the answer token has been committed, while the AR first-token confidence is computed before emitting that token. This self-conditioning asymmetry weakens a causal reading of the cross-paradigm contrast unless commitment-time confidence is also evaluated on a like-for-like basis.

Questions for authors
  1. In Appendix A 8, are candidate score thresholds numerical cutoffs pre-specified independently of calibration data, or empirical quantiles of the calibration scores? If the latter, please give a finite-sample proof that the Clopper–Pearson p-values remain valid for the selected population-conditional risk, or rerun Table 2 with a separately fixed threshold grid. How much does certified coverage change?

  2. How does SEISMO differ empirically and conceptually from the confidence-fluctuation hallucination detector in Song et al ICML 2026 (VGR)? Can a comparable confidence-fluctuation baseline be evaluated on the same diffusion-VLM QA captures, and can the first multimodal trajectory framing be revised accordingly?

  3. At the final diffusion step, is the answer position already unmasked and hence included in the model's conditioning context? Please compare endpoint entropy with predictive uncertainty before token commitment, and with the probability assigned to the committed answer at the actual commitment step, including a paired test against mean step entropy. Is the asserted AR-vs-diffusion inversion still present under matched pre-decision readouts?

  4. The logit-lens analysis reports 85% under-encoding at its default threshold, but the per-layer full-vs-blank causal analysis revises strictly under-encoded cases to about 57%, leaving substantial failed arbitration. Which experiments distinguish absent visual information from present but improperly selected evidence? If the distinction cannot be resolved, should the conclusion that evidence is 'lost before the decoder runs be softened?

5 comments
AuthorsOpenReview ↗

Response to Reviewer ELvu (1/3)

Thank you for your careful review. We ran every check you asked for. We used the released records so anyone can repeat each check on a normal CPU. All main results hold and some get stronger.

Q1. "In Appendix A 8, are candidate score thresholds numerical cutoffs pre-specified independently of calibration data, or empirical quantiles of the calibration scores? If the latter, please give a finite-sample proof that the Clopper–Pearson p-values remain valid for the selected population-conditional risk, or rerun Table 2 with a separately fixed threshold grid. How much does certified coverage change?"

The paper already checks the guarantee on 200 random splits (Table 2 on page 7 and Appendix A.8 on page 20). We have now also rerun Table 2 with a separately fixed grid as you asked. SEISMO stays just as strong under this stricter rule and in the hardest settings it gets even stronger. It still answers up to 8.5 times more questions safely than confidence.

In the paper we chose the cutoff points from the same data we used to set the guarantee. To remove any doubt we now fix the cutoff points first. We take them from a separate set of POPE questions that is never used to set or check the guarantee. With fixed cutoff points the guarantee follows directly from standard theory. Then we ran the full test again 200 times.

Share of POPE questions each method can answer safely (1200 questions)

Allowed errorConfidenceSEISMO
10%6%51%
15%18%79%
20%32%94%

What this shows

  • SEISMO is as strong as in the paper. No result drops by more than 2 points. In two of the hardest settings SEISMO goes up (47% to 51% and 21% to 27%).
  • SEISMO still answers far more questions safely than confidence.
  • For SEISMO the promise holds. In more than 91% of runs the error stays under the allowed level. The rule asks for 90%.
  • A random score answers almost no questions. So the test is fair.
  • With 400 questions the picture is the same. SEISMO answers 27% and 81% at 15% and 20% allowed error. Confidence answers 7% and 24%.
  • We also ran an even stricter check on 400 questions that needs no extra assumptions. SEISMO still answers 58% at 15% allowed error. Confidence answers only 1%.

The guarantee method is standard. Our score is what makes it useful.

AuthorsOpenReview ↗

Response to Reviewer ELvu (2/3)

Q2. "How does SEISMO differ empirically and conceptually from the confidence-fluctuation hallucination detector in Song et al ICML 2026 (VGR)? Can a comparable confidence-fluctuation baseline be evaluated on the same diffusion-VLM QA captures, and can the first multimodal trajectory framing be revised accordingly?"

Thank you for the pointer. We read VGR closely and we will cite it. VGR and SEISMO are very different papers. They ask a different question and use a different signal. They give a different explanation and work on a different task. They also produce a different output. The table shows that they differ on every point.

VGRSEISMO
The questionHow do we fix wrong words in a caption?When can we trust an answer at all?
The signalHow much the top probability jumps up and downHow spread out the whole answer distribution is averaged over all steps
Why confidence failsLanguage habits compete with what the image shows (a theory)The decoding rule itself erases the doubt (proven by changing the rule)
The taskLong image captionsShort answer questions on six benchmarks
The outputA rewritten captionA trust score and a calibrated error chance and a safety guarantee
What it needsFine-tuning plus an object detector plus a text parserNothing. One normal decoding pass
Extra time2.1 times slowerNo extra time
Detector tested onLLaDA-V onlyThree model families plus a direct comparison with AR models
Acting on the resultRewrite the captionHold back the answer or send it to a backup model (accuracy 0.830 to 0.868)

The two signals also behave very differently. The VGR paper gives its exact score (their Equation 3). We ran it on our step by step records next to SEISMO (300 or 400 questions per test). A higher number means the score is better at finding wrong answers.

ScoreLLaDA-V POPELLaDA-V GQAMMaDA POPEMMaDA GQA
SEISMO0.760.720.650.71
VGR score (exact formula)0.680.520.660.56

SEISMO is significantly better in three of the four tests (by 0.08 and 0.21 and 0.15). The fourth is a tie. On LLaDA-V GQA the VGR score is close to a coin flip (0.52) while SEISMO reaches 0.72. So SEISMO is not a version of the VGR signal. It measures something else and it finds errors that the VGR signal misses.

To our knowledge SEISMO is the first framework that turns the step signal of a diffusion VLM into an error guarantee. Section 2 (page 4) of our paper already sets SEISMO apart from the closest step based detectors (TraceDet and DynHD and HIVE).

In short VGR is a tool for editing captions. SEISMO is a framework for deciding when a diffusion VLM can be trusted.

AuthorsOpenReview ↗

Response to Reviewer ELvu (3/3)

Q3. "At the final diffusion step, is the answer position already unmasked and hence included in the model's conditioning context? Please compare endpoint entropy with predictive uncertainty before token commitment, and with the probability assigned to the committed answer at the actual commitment step, including a paired test against mean step entropy. Is the asserted AR-vs-diffusion inversion still present under matched pre-decision readouts?"

Thank you for raising this important point. The answer is yes. At the final step the model has already placed its answer and can see it. This is exactly what our paper studies. The paper already shows that the decoding rule is what makes final confidence fail (Section 6 on page 8). Changing only the decoding rule raises confidence from 0.48 to 0.56. Table 10 (page 26) already shows that the first steps hold the signal.

Now we also read confidence at every point you asked about. An AR model writes its answer in one go. So its confidence is read at the moment it commits to the answer. The fair match in a diffusion model is the moment it places the answer word. At that moment the model has not seen the word yet. All numbers in the table use the same questions as Table 1.

Where we read confidence (LLaDA-V)Has the model seen its own answer here?POPEGQA
The moment the answer word is placed (the fair match to AR)No0.550.58
After decoding with the answer hidden againOnly the words around it0.520.41
Final step (what other methods use)Yes0.500.37
SEISMO scoreReads every step0.780.72
AR model (Qwen2-VL) before it writes the answerNo0.72 to 0.770.74 to 0.76

What this shows

  • The gap with AR models stays under the fair reading. At the moment it places the answer the diffusion model's confidence is only 0.55 and 0.58. The AR model's confidence is 0.72 to 0.77. So the problem is not caused by an unfair comparison.
  • SEISMO still wins under the fair reading. It is better by 0.23 on POPE and by 0.14 on GQA. Both gaps are significant in a paired test.
  • Confidence gets worse the more of its own text the model sees. On GQA it falls from 0.58 to 0.41 to 0.37. The lowest point is the final step. That is the reading every real system uses.
  • The doubt is there before the model commits. At step 1 the model's uncertainty already flags errors (0.69 on POPE and 0.76 on GQA). Decoding then wipes this doubt out of the final answer.

This is the core finding of SEISMO shown step by step. In a diffusion VLM the decoding process itself hides the model's doubt. SEISMO reads that doubt back from the steps the model already runs.

Q4. "The logit-lens analysis reports 85% under-encoding at its default threshold, but the per-layer full-vs-blank causal analysis revises strictly under-encoded cases to about 57%, leaving substantial failed arbitration. Which experiments distinguish absent visual information from present but improperly selected evidence? If the distinction cannot be resolved, should the conclusion that evidence is 'lost before the decoder runs be softened?"

Thank you for this careful question. The paper already answers it with four tests and reports both numbers side by side (Section 6 on pages 8 and 9 and Appendix E.1 on pages 31 to 33).

TestWhat we foundWhat it tells us
Remove the imageAccuracy drops from 0.835 to 0.467The model really uses the image
Look inside every layer (97 wrong answers)The right answer never comes out on top in 85% of cases (and in more than half at every cutoff we tried)In most errors the right answer is never strong enough
Swap in a blank image at each layerAbout 57% show no visual evidence at allThe rest have some evidence but it loses to other signals
Probe the image features directlyThe image does not beat a text only guess (0.715 vs 0.721)Little extra object evidence reaches the language model

Appendix E.1 (page 33) already draws the safe conclusion. Both cases lead to the same place. In most wrong answers the right answer is never the model's top choice at any layer before decoding. It does not matter whether the evidence is missing or loses an inner contest. A decoding rule has nothing good to pick from. This is why every fix we tried at decoding time did not help. It is also why SEISMO is built to detect and act. Sending flagged questions to an AR backup raises accuracy from 0.830 to 0.868. This matches replacing the model completely while using the backup for only one in five questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the rebuttal and revision

We sincerely thank all three reviewers for their careful reading and valuable feedback. Their comments made the paper much stronger. We have uploaded a revised PDF. The new results and the main changes are summarized below.

QuestionReviewersAnswer
Is the paper clear, and are the motivation, method and contributions well linked?kLn1, MkdXYes. We made a major revision of the whole paper. It now follows one central message and five questions Q1 to Q5. Section 3.1 shows how the method follows from the problem, and each results section ends with an answer that leads to the next question (Sections 1, 3.1 and 5, Table 1)
Is the Learn-then-Test guarantee valid when the cut-offs come from the same data?ELvuGuard now fixes its cut-offs in advance from a separate pool of POPE questions. At error budgets of 10%, 15% and 20% SEISMO answers 51%, 79% and 94% of the questions. The model's own confidence answers only 6%, 18% and 32%. The promise holds in at least 91% of 200 random splits (Section 3.3, Table 8, Appendix R.1)
How does SEISMO differ from VGR?ELvuVGR edits captions while SEISMO decides when an answer can be trusted and gives an error guarantee. The exact VGR score is weaker in three of four tests (0.76 against 0.68, 0.72 against 0.52 and 0.71 against 0.56) (Section 2, Table 7, Appendix R.3)
Is the last-step confidence a fair test, since the model already sees its answer?ELvuAt the round where the answer is filled in, the fair match to an AR model, the confidence stays weak (0.55 and 0.58) while the AR model reaches 0.72 to 0.77. SEISMO beats this fair reading by 0.23 and 0.14 with paired intervals above zero (Section 5.1, Table 5, Appendix R.2)
Is the visual evidence missing or outvoted?ELvuIn most wrong answers the right word is never on top at any layer before decoding. So no decoding rule can pick it, which is why SEISMO detects and routes. We now state clearly that our tests cannot yet tell the two cases apart (Section 5.5, Appendices E.1 and R.8)
Does the gain hold beyond LLaDA models?kLn1, MkdXYes. One simple rule explains every model across three families. On Nemotron, which is not built on LLaDA, SEISMO lifts detection from 0.51 to 0.82. SEISMO wins 5 of 9 new cells and ties the other 4 (Section 5.2, Table 6, Appendix R.5)
Is entropy needed, or would the top probability do?kLn1Reading all rounds is what matters. Both beat the last round by about 0.3 and entropy is slightly better on open GQA questions (Section 5.3, Table 7, Appendix R.4)
Does any signal remain in open-ended text after removing the commit step?kLn1The remaining signal sits in words filled in late (0.65 to 0.70) (Section 5.5, Appendix R.7)
How do Dream models differ from LLaDA?kLn1On LLaDA-V the doubt shrinks about 830 times from the first round to the last. Dream models lose very little, so their last round still holds the doubt (Section 5.2, Appendix R.6)
Can the system warn without labels?kLn1Yes. A new label-free alarm caught all four data shifts in every run after a median of 5 to 211 questions. It rang in at most 1% of runs where the promise held (Sections 3.3 and 5.4, Appendix R.9)
Are the method and its variables defined in the main text?MkdXYes. Every signal with its formula is in Table 3. Section 3 now defines the score, Cal, Guard, both alarms and the full TCR rule (Sections 3.2 to 3.4)
Are the claims too broad?MkdXEvery claim now comes with the evidence behind it. The value of the guarantee comes from the SEISMO score, and the main finding holds across three model families (Abstract, Sections 5.2 and 5.4)
Is AUROC the same as the Mann–Whitney statistic?MkdXYes. AUROC = U/(n1 n0) and a numerical check gives the same value three ways (Section 3.2, Appendix R.10)
What does the e-process monitor do?MkdXIt keeps a running betting score and rings when the error rises. Its false-alarm rate is 4.5% and it rings in every run when the error rises to 35% (Section 3.3, Appendix R.9)

Changes in the revised PDF

  • A major revision of the whole paper for clarity. It now follows one central message and five questions Q1 to Q5. Table 1 gives each short answer and each results section ends with an "Answer to Qn" line (Sections 1 and 5).
  • Section 2 and Table 2: expanded related work, including VGR and the venue of each work.
  • Section 3: a self-contained method with all signals (Table 3), the SEISMO score (Eq. 1), Guard, both alarms and the full TCR rule (Eq. 2).
  • Section 4: eight diffusion VLMs from three model families and eight data sets.
  • Section 5: each question now has its own main table (Tables 4 to 9), including the fair readout (Table 5), the three families (Table 6), VGR and the ablation (Table 7), fixed cut-offs (Table 8) and routing (Table 9).

We are grateful for the reviewers' guidance and we are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.