acceptodds
Withdrawn from ICLR 2027

Pairwise Agreement Does Not Certify a Shared World

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models can answer questions about a scene correctly while their answer probabilities fail to describe one shared world. We measure this failure by comparing the smallest change needed to make overlapping marginals agree with the smallest change needed to obtain them from one joint distribution. The difference isolates a global inconsistency that pairwise checks omit; a third term measures the excess discrepancy of the model's global readout. Probabilities are computed for complete autoregressive answers, including the native termination token. Across five models, all four with complete eight-world competence coverage exhibit a positive gap, and three reproduce it under both prompt forms. Testing three models on 512 GQA and CLEVR images yields 32 positive conditions on 30 distinct images with correct baseline single-question and joint answers, including 14 images for Llama-based Idefics3. Across four fixed readout variants, answer wording and order can change the global component even when every tested answer remains correct. These findings show why global compatibility must be tested at the probability level, with the answer interface explicitly specified.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Probabilistic Consistency, Marginal Compatibility, Model Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gqA5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a probability-consistency diagnostic for VLMs based on the finite marginal-extension problem. For three pairwise contexts over three binary variables, the authors compute the L1 distance from the local distributions to the overlap polytope, where shared singleton marginals agree, and to the marginal polytope, where all local distributions are induced by a common eight-state joint distribution. Their difference, B_pair-to-world, measures the global incompatibility missed by pairwise consistency checks. Model probabilities are obtained from teacher-forced complete autoregressive candidate answers, including the native termination token, and normalized within the specified candidate family. The controlled study covers five VLMs, four evidence carriers, and two prompt forms; all four models satisfying the eight-world competence-coverage criterion exhibit a positive gap. An extension to 512 GQA/CLEVR images identifies 30 images for which the single-question and joint answers are correct but at least one readout has a positive gap.

Critical strengths
  1. The diagnostic has a precise and verifiable mathematical meaning, which is impressive.

  2. The probability-event definition and numerical reproducibility are rigorous, and authors score complete candidate sequences with each model's native termination token rather than using prefix probabilities alone.

  3. The paper carefully distinguishes existence, competence coverage, and readout stability.

Critical weaknesses
  1. The measurement depends strongly on a closed candidate family and a single surface string per state, so it may not represent the model's total probability for the semantic event.

  2. The empirical gap is not stable across answer interfaces, which limits any representation-level interpretation. A more accurate claim is that some specified readouts cannot be jointly extended. This qualification should be made especially prominent near the title, abstract, and main conclusion, and the paper should seek an aggregation that remains stable across semantically equivalent interfaces.

  3. Positive cases are relatively infrequent and some gaps are numerically small, so the current experiments establish existence rather than prevalence or practical severity.

  4. The diagnostic has no direct demonstrated connection to downstream decision loss, and the visual extension changed its execution protocol after the development qualification failed.

Questions for authors
  1. If multiple equivalent answer templates are enumerated for every semantic state and their complete-sequence probabilities are summed, or if separate distributions are constructed across several templates, how many controlled and natural-image positive cases remain positive? Can this distinguish surface-form competition from more stable semantic incompatibility?

  2. Does B_pair-to-world itself, without additional world-distance or violation features, predict any held-out downstream error, overconfidence, or planning regret after controlling for delta_pair, entropy, and answer correctness? If not, what concrete decision should an evaluator make based on this diagnostic?

Reviewer_oEMWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether the answer probabilities of vision-language models (VLMs) for separately queried local questions about a scene can be explained by one joint distribution over the scene. Using three binary variables (door, key, hazard), the model is queried for three pairwise laws and one global law. The authors measure the L1 distance from the pairwise laws to the overlap polytope (matching singleton marginals) and to the marginal polytope (laws induced by a joint distribution), and define the pair-to-world gap as the difference.

Critical strengths
  1. The question and formulation are really interesting: whether a model's separately queried local answers describe one shared world goes beyond answer accuracy and pairwise consistency.
  2. The topic is relevant for multi-view, multi-agent and world-model settings. I especially like Figure 1 which is intuitive for understanding.
  3. The decomposition into overlap distance, pair-to-world gap and global-readout excess, all in the same L1 units, is clean and easy to compute with linear programs.
Critical weaknesses
  1. Only the minimal structure is studied. All experiments, including the natural-image extension, use three binary variables forming the smallest cycle (2x2x2). This is a great start point, but real scenes involve more variables, more states per variable and more complex cycle structures. It is unclear whether the pair-to-world gap grows (more cycles and more global constraints) or becomes negligible relative to pairwise mismatch in larger settings, which directly determines whether the proposed diagnostic is useful in practice. The method also seems hard to scale? The global query enumerates all joint candidates (2^n for n binary variables), and the eligibility criterion requires the model to recognize every world. More explanations and experiments here would make the paper for interesting.
  2. Prompts here are very essential, but this is not studied systematically. In the controlled study, only two prompt forms are used, and they give very different results. There is no analysis and no noise baseline from prompt variation. Since each local law comes from a separate query with a different prompt, part or all of the measured gap may be a property of the prompts rather than of the model's representation of the scene. The paper treats prompt dependence as a reporting requirement ("report the answer interface") rather than as a possible confound for the main claim. But is this proper? why?
  3. The criterion for a positive gap is a numerical tolerance, but this should be model specific no? Does it make sense to have one tolerance threshold for all models? This depends on the uncertainty of a model no? Many reported positives are tiny and model-side variability (e.g., bf16 inference, prompt variation) is not quantified. Statements such as "every qualified model exposes the omitted global component" therefore overstate the evidence.
Questions for authors
  1. How does the pair-to-world gap behave with more variables, more states per variable or more complex cycle structures? Can the method be applied without a global query that enumerates all joint candidates? How scalable is the method?
  2. Can the authors estimate the variability of the gap under paraphrased prompts and report only gaps exceeding this noise level? Does the gap persist if probabilities are averaged over a family of prompts? How sensitive are the results to different prompts?
  3. Can the authors report an effect-size criterion (e.g., gap as a fraction of the world distance) in addition to the 10^-8 numerical criterion, and show how many positives remain? Is this threshold really model specific? How does it take into consideration the different sampling uncertainty of different models? Can it be considered?
  4. Is there an example where the global gap, rather than pairwise mismatch, leads to an incorrect downstream inference or decision? If so, that would be interesting too.
Reviewer_9Wi7OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper tests whether separately queried VLM answer distributions can be marginals of one joint distribution over a scene. It compares the minimum L1 correction needed for agreement on overlapping variables with the correction needed for global compatibility. Their difference measures the additional cost of requiring a joint distribution. A third term measures how much further the model's separately queried global answer is from the local distributions than a nearest compatible explanation. The experiments use complete answer probabilities, including termination, and find positive gaps in controlled stimuli and in correctly answered GQA and CLEVR conditions. They also show that the gap depends on answer wording and order.

Critical strengths
  • Principled formulation. The paper addresses an important distinction between pairwise consistency and global realizability, providing an interpretable diagnostic beyond answer correctness.
  • Careful experimental design. The use of complete autoregressive probabilities, competence controls, and multiple VLM architectures strengthens the evaluation. The transparent reporting of negative results and readout sensitivity is also appreciated.
Critical weaknesses
  1. Limited novelty and comparative validation. The proposed framework substantially overlaps with the coherence-distance approach of Kotawala [1] in its assessment of global probabilistic consistency. The paper fails to demonstrate a clear methodological advantage over this prior work, and the lack of matched baseline comparisons or targeted ablations leaves the added value of the proposed decomposition unsubstantiated.
  2. Limited empirical and practical significance. All 187 eligible controlled conditions already exhibit pairwise disagreement, while only 35 show an additional global gap (median 8.56×10−58.56 \times 10^{-5}). In natural-image experiments, only 32/877 strict-correct conditions show positive gaps. Although some gaps are substantial, the limited prevalence and lack of demonstrated predictive value beyond pairwise measures weaken the practical significance.
  3. Substantial sensitivity to answer interfaces. Among 119 matched model-image cases correct across all four readout variants, none exhibits a positive gap consistently across all four. This raises concerns about whether the diagnostic captures robust probabilistic inconsistency rather than interface-specific variation.
  4. Restricted candidate normalization. The method normalizes probabilities over predefined answer strings without reporting the total probability mass captured by these candidates. This makes it difficult to determine whether the observed gaps reflect meaningful semantic inconsistencies or artifacts of the answer representation.
[1] Anany Kotawala. Locally Coherent, Globally Incoherent: Bounding Compositional Incoherence in Multi-Component LLM Agents. arXiv:2605.30335, 2026.
Questions for authors
  • What substantive methodological contribution does the proposed decomposition offer beyond Kotawala (2026)? Can the authors provide matched comparisons or targeted ablations demonstrating its added value?
  • Does the global gap provide incremental predictive value for downstream failures beyond existing pairwise disagreement measures?
  • Given the substantial readout sensitivity, can the authors evaluate whether the observed gaps persist across semantically equivalent answer paraphrases? How should practitioners interpret the lack of consistency across existing readouts?
  • How much probability mass is captured by the predefined candidate sets, and how sensitive are the findings to this normalization?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.