- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a probability-consistency diagnostic for VLMs based on the finite marginal-extension problem. For three pairwise contexts over three binary variables, the authors compute the L1 distance from the local distributions to the overlap polytope, where shared singleton marginals agree, and to the marginal polytope, where all local distributions are induced by a common eight-state joint distribution. Their difference, B_pair-to-world, measures the global incompatibility missed by pairwise consistency checks. Model probabilities are obtained from teacher-forced complete autoregressive candidate answers, including the native termination token, and normalized within the specified candidate family. The controlled study covers five VLMs, four evidence carriers, and two prompt forms; all four models satisfying the eight-world competence-coverage criterion exhibit a positive gap. An extension to 512 GQA/CLEVR images identifies 30 images for which the single-question and joint answers are correct but at least one readout has a positive gap.
Critical strengths
-
The diagnostic has a precise and verifiable mathematical meaning, which is impressive.
-
The probability-event definition and numerical reproducibility are rigorous, and authors score complete candidate sequences with each model's native termination token rather than using prefix probabilities alone.
-
The paper carefully distinguishes existence, competence coverage, and readout stability.
Critical weaknesses
-
The measurement depends strongly on a closed candidate family and a single surface string per state, so it may not represent the model's total probability for the semantic event.
-
The empirical gap is not stable across answer interfaces, which limits any representation-level interpretation. A more accurate claim is that some specified readouts cannot be jointly extended. This qualification should be made especially prominent near the title, abstract, and main conclusion, and the paper should seek an aggregation that remains stable across semantically equivalent interfaces.
-
Positive cases are relatively infrequent and some gaps are numerically small, so the current experiments establish existence rather than prevalence or practical severity.
-
The diagnostic has no direct demonstrated connection to downstream decision loss, and the visual extension changed its execution protocol after the development qualification failed.
Questions for authors
-
If multiple equivalent answer templates are enumerated for every semantic state and their complete-sequence probabilities are summed, or if separate distributions are constructed across several templates, how many controlled and natural-image positive cases remain positive? Can this distinguish surface-form competition from more stable semantic incompatibility?
-
Does B_pair-to-world itself, without additional world-distance or violation features, predict any held-out downstream error, overconfidence, or planning regret after controlling for delta_pair, entropy, and answer correctness? If not, what concrete decision should an evaluator make based on this diagnostic?