acceptodds
Under review as a conference paper at ICLR 2027

Visual Overinterpretation in the Prefill Representations of Vision–Language Models

Abstract

Vision–Language Models (VLMs) have advanced rapidly in recent years, but they still suffer from hallucinations. Prior studies have identified various causes of hallucination and developed mitigation methods, primarily focusing on the generation process. More recent work moves upstream, revealing that hallucination risk is already encoded in pre-generation representations and can be reduced through prefill-time interventions. Nevertheless, these studies characterize hallucination risk through latent token-level or representation-level signals, lacking semantic interpretability of what kinds of concepts are easily misrepresented and how such biases emerge during prefill. Our work investigates this problem from a new concept-level perspective. Specifically, we formulate a fine-grained plausibility-aware concept system that classifies risk concepts into semantically meaningful categories. This concept system includes plausible absent concepts (PACs) and implausible absent concepts (imPACs), with PACs further distinguished by local visual resemblance, global scene semantics, or both. Based on this, we introduce PAC, a manually annotated benchmark for controlled concept-level analysis. We further design a dual-contrast prefill analysis framework that examines PAC-imPAC preference at both behavioral and representational levels and traces its evolution across prefill layers. Experiments across multiple VLMs consistently show that PACs receive stronger support than matched imPACs at both the behavioral and representational levels. Importantly, this preference emerges in intermediate prefill layers. We term this phenomenon visual overinterpretation, providing a concept-level semantic interpretation of prefill hallucination risk. Experiments in open-ended generation further show that this phenomenon is associated with subsequent object hallucinations.

Primary area:
interpretability and explainable AI
Keywords:
Vision–Language Models, Hallucination, Visual Overinterpretation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cCuXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates hallucination risk in LVLMs from the perspective of prefill representations. It introduces a plausibility-aware concept taxonomy that distinguishes plausible absent concepts (PACs) from implausible absent concepts (imPACs), and constructs the PAC³ benchmark for controlled analysis. The authors compare PACs and matched imPACs at the behavioral, representation, and generation levels, and report that PACs are more likely to be falsely recognized, receive stronger image-conditioned support in intermediate prefill layers, and are more likely to appear in subsequent generations. They refer to this phenomenon as “visual overinterpretation” and argue that it provides a concept-level interpretation of pre-generation hallucination risk.

Critical strengths

The paper studies an interesting and relatively underexplored question regarding hallucination risk before generation. Focusing on prefill representations provides a potentially useful perspective complementary to analyses centered on decoding behavior.

Critical weaknesses

The paper does not clearly distinguish “visual overinterpretation” from normal context-conditioned semantic inference. Since PACs are defined precisely as absent concepts that are plausible given the image context, it is expected that a VLM assigns them higher support than implausible absent concepts. The reported ordering S(c_g)>S(c_p)>S(c_n) can therefore be interpreted as a reasonable evidence hierarchy rather than a hallucination-specific failure mode.

The interpretation of intermediate-layer logits as “concept-level representations” is insufficiently justified. The proposed concept support is essentially a Yes–No logit margin obtained by projecting the final prompt representation at each layer. A larger intermediate-layer logit for “Yes” does not necessarily imply that the corresponding concept is explicitly represented or misrepresented internally, leaving a substantial gap between the measurement and the claimed representational interpretation.

The visual-null baseline is central to the method but is not adequately validated. The entire POG analysis relies on subtracting a model-specific visual-null response from the image-conditioned response. However, removing or replacing visual input can itself induce architecture-dependent distribution shifts. Without careful ablations over alternative null constructions, it is unclear whether the measured gap reflects visual evidence or artifacts introduced by the chosen baseline.

The PAC–imPAC matching procedure does not sufficiently control semantic confounders. Matching is mainly based on concept frequency and token length, while PACs and imPACs may still differ systematically in semantic category, lexical familiarity, visual recognizability, or prompt difficulty. Therefore, the observed PAC–imPAC gap cannot be cleanly attributed to plausibility alone.

The evidence connecting prefill overinterpretation to downstream hallucination is weak. The paper establishes correlation rather than causation, and the association is strongly model dependent. In particular, Qwen3-VL shows only a weak POG–GOG correlation, while many PAC–imPAC pairs have zero generation difference. This substantially weakens the claim that the identified prefill phenomenon is a meaningful precursor of hallucination.

The current manuscript has significant reproducibility and completeness issues. Appendix A claims to provide the benchmark construction, prompts, model-specific visual-null implementations, discovery/confirmation split, and statistical procedures, but these details are not actually included in the provided manuscript. These missing details concern several core components of the proposed analysis and make the main results difficult to independently assess or reproduce.

Questions for authors

See weaknesses.

1 comment

Thank you for your valuable feedback. We have uploaded the revised manuscript (changes in blue) and supplementary materials with additional experiments. Our responses are as follows:

1. Visual Overinterpretation vs. Normal Semantic Inference.

We acknowledge that higher support for PACs may reflect reasonable contextual inference. However, our findings extend beyond such preferences. Visual overinterpretation encompasses two complementary levels:

  • Behavioral (RQ1): Both PACs and imPACs are verified as absent, yet PACs elicit significantly more false-positive judgments across all models (Table 2), demonstrating that plausibility-related preferences can manifest as factual errors.
  • Prefill representational (RQ2): We characterize how preferential support for these error-prone concepts emerges and evolves across layers. Although higher support is not itself a factual error, its association with downstream hallucinations (RQ3) highlights its relevance to hallucination susceptibility. Behavioral gaps persist under lexical and semantic controls (Appendices C.2 and G).

2. Interpretation of Intermediate-Layer Concept Support.

Following layer-wise probing studies [1,2], we use concept-specific queries and binary logit readouts to probe existence judgments across layers (Sec. 3.3). Concept-level interpretability arises from semantically defined queries and controlled contrasts, rather than assuming explicit concept encodings. This approach reveals model-dependent layer-wise dynamics (Finding 2), while downstream associations (Finding 3) support its behavioral relevance (Sec. 4.3). [1] Wang et al. Same Attention, Different Truths: Put Logit-Lens over Visual Attention to Detect and Mitigate LVLM Object Hallucination. CVPR, 2026. [2] Belrose et al. Eliciting Latent Predictions from Transformers with the Tuned Lens. arXiv:2303.08112, 2023.

3. Robustness to Alternative Visual Controls.

Our visual-null implementations preserve text, sequence length, and visual-token positions (Sec. 4.1.2). Under additional blank and unrelated natural image controls, onset, peak, final-layer, and all-layer support gaps remain positive across models, with all 95% CIs excluding zero (Appendix D.3, Tables 14–15), suggesting that preferential PAC support is not specific to our visual-null implementation.

4. PAC–imPAC Matching and Semantic Confounders.

PACs and imPACs are matched within the same image, with lexical frequency and token length considered during benchmark construction. To further address potential confounders, we conducted three additional analyses:

  • Lexical control: Adjusted gaps remain close to raw gaps after controlling for frequency and token length (Appendix C.2, Table 9).
  • Semantic control: Controlling for co-occurrence log-lift or co-occurrence-vector cosine similarity on all 313 triplets, behavioral gaps remain positive across models, with all 95% CIs excluding zero (Appendix G, Table 19).
  • Prompt robustness: Positive behavioral gaps persist across alternative formulations (Appendix C.3, Table 10). Together, these results suggest that the observed behavioral preference cannot be fully explained by the measured lexical and semantic factors or specific prompt formulations. We acknowledge that other factors, such as concept-specific visual recognizability, may still contribute.

5. Association with Downstream Hallucinations.

Our revised analysis includes all 313 pairs (186 images), including zero-gap pairs (55.9%/82.7%/95.2%), yielding correlations of 0.494/0.433/0.191 (Table 4). After text-only prior adjustment, associations remain significantly positive in LLaVA and DeepSeek-VL but are inconclusive in Qwen3-VL. Onset-layer support shows no reliable incremental predictive value beyond final-layer signals. Importantly, Findings 3–4 motivate examining prefill and decoding separately. Qwen3-VL retains substantial prefill support gaps, yet its low PAC (1.25%) and grounded concept (44.41%) mention rates suggest that conservative decoding may suppress hallucinations at the cost of concept coverage (Sec. 4.3, Table 6). Thus, fewer hallucinated PAC mentions do not necessarily imply reduced prefill susceptibility. Our contribution is to characterize concept-level prefill susceptibility that output-level evaluation may overlook, rather than establish a universal predictor or causal mechanism (Appendix F).

6. Reproducibility and Completeness.

We apologize for the incomplete appendix. The revision details benchmark construction (Appendix A.1), prompts and scoring (A.2, C.3), visual-null implementations and controls (Sec. 4.1.2, Appendix D.3), image-disjoint splits and layer selection (D.2), and statistical procedures (A.3). We also consolidate experimental observations into four key findings and discuss their implications and limitations (Sec. 4.3, Appendix F).

Reviewer_rAwWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether vision-language models encode biases toward plausible but visually absent concepts before generation. The authors introduce a plausibility-aware concept taxonomy that separates plausible absent concepts (PACs) from matched implausible absent concepts (imPACs), and construct PAC3, a manually annotated benchmark for controlled analysis. Using behavioral and representation-level contrasts, the paper studies PAC–imPAC preference during prefill and its association with subsequent hallucinated generation. Experiments across three VLMs show that PACs are more likely to be falsely recognized, receive stronger image-conditioned representation support, and correlate with downstream object hallucination.

Critical strengths
  1. The paper studies whether hallucination-related bias exists before generation and whether such bias can be interpreted semantically rather than only through latent features. The PAC/imPAC contrast provides a more interpretable unit of analysis than generic hidden-state probing.

  2. Matching absent concepts by frequency and token length while controlling for the same image allows the authors to separate plausibility effects from simple concept-level differences.

  3. The experiments cover multiple aspects of the phenomenon, including behavioral recognition, layer-wise representation analysis, and open-ended generation association across multiple VLMs.

Critical weaknesses
  1. The main limitation is that the paper establishes association rather than a mechanism for hallucination. PACs receiving stronger prefill support and being generated more frequently does not demonstrate that this representation bias causes hallucination; the same underlying semantic prior could influence both the measured support and the final output.

  2. The PAC/imPAC construction may not fully isolate plausibility from semantic familiarity. Although the paper controls frequency and token length, plausible concepts are inherently more related to the image context, language distribution, and object co-occurrence statistics. More analysis controlling for embedding similarity, semantic relatedness, or language-model prior probability would strengthen the causal interpretation of the contrast.

  3. The representation analysis depends on visual-null subtraction and logit-based concept support, but the meaning of this quantity is not sufficiently validated. It is unclear whether higher image-conditioned support corresponds to stronger visual grounding or simply stronger interaction between visual features and existing language priors.

  4. The downstream hallucination analysis uses correlation between POG and GOG, but the strength varies substantially across models. In particular, Qwen3-VL-8B shows only a weak association (ρ=0.17), suggesting that the relationship between prefill overinterpretation and generation hallucination may not be a general phenomenon. The paper should better discuss when the proposed phenomenon is predictive and when it is not.

  5. The benchmark is manually constructed and relatively small (453 controlled instances). While this is sufficient for controlled analysis, the paper makes broad claims about VLM hallucination behavior, and additional validation on larger automatically constructed concept sets or more diverse image distributions would improve confidence.

Questions for authors

see weakness

1 comment

Thank you for your valuable feedback. We have revised the manuscript and conducted additional experiments addressing your concerns. Our responses are as follows:

1. Key Findings and Causal Interpretation.

We agree that association alone does not establish causality. Our contribution is to provide a concept-level interpretation of prefill hallucination risk. Visual overinterpretation can coexist with strong recognition performance (Finding 1), emerges around the network midpoint (Finding 2), and can persist despite fewer hallucinated PAC mentions (Finding 4). These findings reveal concept-level susceptibility that may be obscured by aggregate recognition and generation metrics, while identifying candidate layers for future causal interventions (Sec. 4.3, Appendix F).

2. Plausibility and Semantic Confounders.

We conducted three additional analyses:

  • Lexical control. Regression-adjusted behavioral gaps remain close to raw gaps, suggesting little influence from residual frequency and token-length differences (Appendix C.2, Table 9).
  • Semantic-relatedness control. Controlling for co-occurrence log-lift or co-occurrence vector cosine similarity on all 313 triplets, behavioral gaps remain positive across all models, with all 95% CIs excluding zero. Changes relative to lexical-only adjustment are small (−1.79 to +1.45 pp; Appendix G, Table 19).
  • Language-prior control. After controlling for text-only margins, prefill–generation correlations remain significantly positive in LLaVA and DeepSeek-VL but are inconclusive in Qwen3-VL. CHAIR AUC improves beyond frequency and text-only priors across all models. These results support the relevance of plausibility-related biases beyond measured lexical and semantic factors.

3. Interpretation of Image-Conditioned Support.

Our dual-contrast framework reduces concept-specific textual preferences (Sec. 3.3): ΔSℓ=[mℓ(I,cp)−mℓ(I∅,cp)]−[mℓ(I,cn)−mℓ(I∅,cn)].\Delta S_\ell=[m_\ell(I,c_p)-m_\ell(I_\varnothing,c_p)]-[m_\ell(I,c_n)-m_\ell(I_\varnothing,c_n)]. This quantifies differential image-conditioned support rather than absolute semantic preferences. Under additional blank and unrelated image controls, all six fixed-onset 95% CIs exclude zero, supporting robustness to alternative visual controls (Appendix D.3, Tables 14–15). Findings 1–2 further highlight its complementary diagnostic value beyond aggregate recognition performance (Sec. 4.3). Image-conditioned support may reflect visual–language prior interactions; our analysis characterizes the resulting concept-level preference rather than disentangling these contributions.

4. Model-dependent Prefill–Generation Association

Our full-sample analysis (313 triplets) yields positive correlations of 0.494/0.433/0.191 (Table 4). Findings 3–4 highlight layer-wise dynamics and a potential coverage–hallucination trade-off (Sec. 4.3). Qwen3-VL exhibits substantial prefill support gaps but the lowest PAC (1.25%) and grounded concept (44.41%) mention rates, suggesting that selective generation may partially explain this discrepancy (Sec. 4.3, Table 6). Additional analyses reveal that:

  • Correlations strengthen around the onset of visual overinterpretation and remain relatively stable thereafter, despite declining support gaps after the peak (Appendix E.1, Fig. 6).
  • Under text-only prior adjustment or the same caption prompt, associations remain positive in LLaVA and DeepSeek-VL but are less consistent in Qwen3-VL.
  • Onset-layer signals provide no established incremental predictive value beyond final-layer signals. Thus, visual overinterpretation provides a model-dependent indicator of hallucination susceptibility rather than a universally reliable predictor.

5. Benchmark Scale and Generalizability

We conducted three complementary analyses:

  • Broader concept types. All six model–task combinations on 59 attribute and 51 relation triplets exhibit positive behavioral gaps (Appendix C.4, Table 11).
  • Larger-scale automatic validation. On 500 CHAIR images, we automatically enumerate absent COCO categories (19,229 confirmation pairs/model). Global contextual plausibility, relevant to Global- and Mixed-PACs, improves AUC beyond frequency and text-only priors across all models (Appendix E.2 and additional analysis).
  • Scalable PAC construction. We develop an automated pipeline involving image sampling, candidate proposal, independent AI verification, and triplet construction. A pilot on 24 new COCO images yields 91 questions and 43 triplets, with approximately 70% passing human verification. Further refinement is left for future work. These analyses broaden the empirical support for our findings and demonstrate the potential for scaling our framework.
Reviewer_JEC9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether vision–language models give additional support to plausible but absent concepts before generation. The authors distinguish plausible absent concepts (PAC) from implausible absent concepts (imPAC), divide PAC into local visual similarity, global scene semantics, and mixed types, and construct the PAC3 dataset. They match concept frequency and tokenization length, then measure image-conditioned support using layerwise Yes–No logit margins with image input relative to a visual-null input, comparing the relative advantage of PAC over imPAC. Experiments on three models show higher false-recognition rates for PAC, differences emerging in some intermediate layers, and positive correlations with concept-hallucination frequency in open-ended generation. The study offers a concept-level semantic interpretation, with its main conclusions remaining correlational.

Critical strengths
  1. Concept matching and visual-null differencing form a meaningful analytical design. The object-existence subset compares two absent concepts within the same image while matching COCO frequency and tokenization length for each model. This is more targeted than comparing arbitrary positive and negative objects. Differencing image input against visual-null input also attempts to control for each concept's language prior. In Table 1, all three models have higher false-recognition rates for PAC than for imPAC; for Qwen3, the rates are 14.4% and 0.6%, respectively. This provides preliminary support for organizing hallucination analysis by semantic plausibility.
  2. Selecting layers on a discovery set and evaluating them on a confirmation set with no overlapping images is a reasonable way to reduce favorable layer-selection bias. Table 2 reports positive POG on the confirmation set, and the open-ended generation experiments track prespecified concepts rather than selecting examples only after generation. The authors also distinguish object triplets, attribute/relation extensions, and retrospective CHAIR analysis, and explicitly limit the downstream link to correlation rather than causation. These boundaries give the approach potential as an interpretability framework.
Critical weaknesses
  1. The core representation measurement does not yet rule out response tendencies and readout choices as alternative explanations. Section 4.1.2 uses the representation at the end of a question containing the queried concept and its Yes–No logit margin, whereas open-ended captioning does not use those concept queries. POG may therefore reflect an affirmative response tendency under a particular question format rather than an independently formed representation of that object before caption generation. The visual-null subtraction in Section 3.3 does not automatically remove changes in position, sequence structure, or hidden-state scale caused by deleting visual tokens. Without calibrated layerwise readouts, intermediate-layer margin magnitudes should also not be directly used to rank capabilities across models. For example, Qwen3 has the smallest POG but the highest PI in Table 1, which does not support a single straightforward ranking of resistance to interference. The authors should specify normalization, readout heads, answer tokens, and each model's visual-null implementation, and add blank-image or unrelated-image controls that preserve visual structure, equivalent question formulations, and checks of affirmative/negative answer encoding. Representation analysis under the same prompt used for open-ended captioning, or incremental prediction controlling for text-only priors and final-layer outputs, would better establish the added value of the prefill semantic interpretation.
  2. Incomplete and inconsistent reporting of dataset construction and downstream statistics prevents verification of the conclusions. Section 4.1.1 states that PAC3 contains 313 object triplets and 140 attribute/relation pairs, totaling 1,219 questions. Table 5 lists only 89 attribute pairs and 21 relation pairs, totaling 110; the missing 30 instances and corresponding 60 questions are unexplained. Appendix A on page 12 provides only an overview rather than the promised annotation procedures, annotator agreement, discovery/confirmation split, visual-null implementation, and statistical procedures. Under Equation 8, the generation-rate gap in Table 3 should equal Rp minus Rn. LLaVA's 12.4% and 0.6% imply 11.8 percentage points, but the reported GOG is 11.0; Qwen3's 1.2% and 0.0% imply approximately 1.2, but 1.5 is reported; and DeepSeek's values imply 3.2, but 2.9 is reported. Rounding to one decimal place cannot explain these discrepancies. If image-cluster weighting or different subsets are used, the estimand, denominators, and weights must be reported consistently. Table 8 also uses different sample counts for generation and OGA, leaving unclear whether this affects correlation estimates when generation events are sparse. These issues directly affect the dataset contribution and downstream evidence and require sample manifests, statistical code, and consistently recomputed results.
Questions for authors
  1. Please provide complete PAC3 counts by source, semantic type, unique image, and discovery/confirmation split, and explain the discrepancy between 140 extension pairs and the 110 pairs in Table 5. How many annotators assessed PAC plausibility, local/global type, and imPAC implausibility? How were disagreements handled, what was the inter-annotator agreement, and how well balanced were the concepts after matching?
  2. Please specify the full computation of layerwise Yes–No margins and each model's visual-null implementation. Are the sign and layerwise trajectory of POG stable relative to blank or unrelated images that preserve sequence structure? Under the same prompt used for open-ended generation, or after controlling for text-only priors and final-layer response tendencies, does the prefill signal still add downstream predictive information?
  3. Why does the GOG in Table 3 differ from the reported Rp minus Rn? Please specify the inclusion rules and unique-image counts for the generation and OGA samples in Table 8, and report correlations across all samples. How are the plausibility scores in CHAIR Table 9 produced, are they frozen on the discovery set, and how are candidate absent concepts constructed? Do the AUC and correlations hold when image splits are maintained and text priors are controlled?
1 comment

Thank you for your valuable feedback. We have uploaded the revised manuscript (changes highlighted in blue) and supplementary materials, including additional experiments addressing your concerns. Our responses are as follows:

1. Dataset and Annotation

(1) PAC$^3$ contains 423 triplets, 1,269 questions, and 290 unique images: 313 object-existence triplets from POPE (186 images), 59 attribute triplets from MME (57 images), and 51 relation triplets from MME/VSR (51 images). RQ1 and RQ3 use all object-existence triplets, while RQ2 uses 216 discovery triplets (129 images) and 97 confirmation triplets (57 disjoint images). Local/Global/Mixed counts are 52/119/45 and 21/54/22, respectively (Table 1, Appendix D.2). (2) Regarding the discrepancy between 140 and 110 extension pairs, the archived inventory confirms 30 additional MME object-existence pairs. The revised extension includes 59 attribute and 51 relation triplets (30 MME position, 21 VSR) with same-image imPACs (Appendix C.4, Table 11). (3) Two annotators verified visual absence, rated plausibility (0: implausible, 1: somewhat plausible, 2: highly plausible), and classified PACs by local visual resemblance, global scene context, or both. imPACs must lack plausible visual or contextual cues. Annotators cross-reviewed labels and resolved disagreements jointly. Pre-consensus labels were not retained, preventing agreement estimation (Appendix A.1, Fig. 7). (4) Object-level imPACs were matched on COCO image frequency and token length, followed by manual verification. Standardized mean differences are 0.190 for log frequency and 0.111/0.048/0.102 for token length (LLaVA/DeepSeek-VL/Qwen3-VL). Regression-adjusted gaps remain close to raw gaps (10.22→10.52, 10.54→10.96, 13.74→12.85 pp; Table 9), suggesting little influence from residual frequency and token-length differences.

2. Prefill Readout and Prediction

(1) At each layer, we apply final normalization and the LM head to the last prompt-token representation. The Yes–No margin is the difference between log-sum-exp scores of deduplicated single-token answer variants (uppercase-initial/lowercase, with/without leading spaces). We zero preprocessed image tensors for LLaVA/Qwen3-VL and visual-position embeddings for DeepSeek-VL. The original-minus-null contrast reduces concept-specific textual preferences, followed by the PAC–imPAC contrast to quantify differential image-conditioned support (Eqs. 6–8, Sec. 4.1.2, Appendix A.2). (2) Blank and unrelated image controls preserve token IDs and visual positions across 97 confirmation triplets. Both reproduce positive support gaps, with all fixed-onset, peak, final-layer, and all-layer 95% CIs above zero, demonstrating robustness to alternative visual controls, although precise peak timing can vary (Appendix D.3, Tables 14–15). (3) Behavioral gaps remain positive across all three question formulations (Appendix C.3, Table 10). Under the same caption prompt, fixed-onset correlations are 0.340/0.313/−0.207 on 54 held-out images, suggesting model-dependent alignment. We interpret uncalibrated layer-wise margins within each model rather than directly ranking models. (4) Ridge logistic predictors fitted on 216 discovery triplets predict any PAC mention across ten captions on 97 image-disjoint confirmation triplets. Adding onset-layer support to final-layer image and visual-null margins yields Brier improvements of 0.00099/0.00784/−0.00100, with all 95% CIs including zero. Thus, incremental predictive value beyond final-layer signals remains unestablished.

3. Generation Metrics

(1) GOG (now ΔR\Delta R) averages paired mention-rate gaps within each image and then across images, whereas RpR_p and RnR_n weight triplets equally (Eq. 11, Table 4). (2) We replace the original subset analyses with full-sample generation and correlation analyses covering all 313 triplets (186 images, 10 captions/image), including zero mention-rate gaps. All-layer mean support gaps correlate with mention-rate gaps, yielding Spearman's ρ\rho of 0.494/0.433/0.191 (Table 4, Appendix E.1). (3) CHAIR candidates are COCO categories absent from image annotations, scored by maximum smoothed co-occurrence log-lift with present categories (COCO train2014), capturing global plausibility cues shared by Global- and Mixed-PACs. Quintile thresholds and predictors are fixed on 250 discovery images and evaluated on 250 disjoint confirmation images (19,229 pairs/model; Appendix E.2). (4) After controlling for text-only margins using discovery-fitted regression, correlations on 97 confirmation triplets (57 disjoint images) are 0.546/0.294/0.035, remaining significant in LLaVA and DeepSeek-VL. On CHAIR, adding global contextual plausibility to frequency and text-only priors improves confirmation AUC from 0.793/0.736/0.699 to 0.903/0.819/0.760, with all gain CIs excluding zero. These results support the relevance of plausibility-related biases beyond measured language priors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.