acceptodds
Under review as a conference paper at ICLR 2027

Know More but Say Less: Bridging the Generation–Discrimination Gap of VLMs via Latent-Guided Verification

Abstract

Vision-Language Models (VLMs) can turn unreliable visual inputs into fluent descriptions that downstream systems accept as facts, compromising reasoning and decisions. We attribute this failure to the generation-discrimination (G-D) gap, in which unreliability is omitted during descriptive generation but recognized under explicit verification. Yet its mechanisms remain unclear, and conventional hallucination mitigation through direct internal modification is ill-suited to addressing it. We therefore analyze hidden activations, visual attention, and output distributions. Unreliability evidence remains linearly separable in intermediate representations and visual tokens receive substantial attention, but this distinction is weakly reflected in output probabilities and generated text. Based on these findings, we propose Bridge, which estimates unreliability from prefix activations and selectively injects hierarchical verification prompts without updating VLM parameters or directly modifying internal states. Across four benchmarks, Bridge outperforms all VLM-based baselines on the primary reliability metric. Relative to descriptive generation, it improves F1 by up to 0.845 and Recall by 0.550. On our self-designed 600-sample evaluation across three tasks, it achieves 84.2% task success and reduces unreliable information propagation by 11.7 percentage points, balancing reliability and utility. Our implementation is available at https://anonymous.4open.science/r/GDBRIDGE-F7CD.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Vision-language models, generation-discrimination gap, visual reliability, inference-time intervention

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tzkPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work investigate a generation-discrimination gap issue, i.e., VLM can describe counterfactual image without stating that it is unreliable. The paper first anaylsizes hidden states, attention and output probabilities and then propose BRIDGE to train a small directional projector to select verification prompts of different strengths. Empirically, BRIDGE improves recognition of unrealiable input while increasing inference cost.

Critical strengths
  1. This paper focuses on a practically meaningful yet under studied problem. A description can faithfully describe an unreliable image without acknowledging it is fake or counterfactual. So this study could be helpful for high stack system.

  2. The paper presents deliberate ablation, showcasing meaningful improvement on several benchmarks.

  3. The crafted method itself is simple and intuitive.

Critical weaknesses
  1. Different model or weights will require retraining of proposed projector, making the method less portable and generalizable. It also require white-box model, making it unusable for blackbox API which is the most powerful and suitable model for high stack scenarios.

  2. Benefit of BRIDGE’s routing design is insufficiently isolated.PrivGate [1] already uses latent risk signals to trigger reasoning selectively, and ReasoningGuard [2] already inserts reflection prompts after attention sinks. The existing ablations do not replace a simple probe using the same prefix features and intervention policy.

  3. BRIDGE induces substantial inference cost, and can up to 2x higher latency. The paper reports this overhead, but lacks an end-to-end reliability/cost comparison with simpler prompting and routing alternatives.

[1] https://icml.cc/virtual/2026/poster/61631 [2] https://aclanthology.org/2026.acl-long.1453/

Questions for authors
  1. Does Figure 3d’s conclusion survive equal-sized within/between-class comparisons, a label-permutation null, and analysis of full output vectors? Which datasets, class counts, and scene-group splits underlie each diagnostic plot?

  2. How do linear and standard MLP probes perform with the same layer, prefix features, and prompt-insertion policy as BRIDGE? Can binary versus hierarchical routing be compared while holding the detector fixed?

8 comments
AuthorsOpenReview ↗

Response 1

Comment 1:

"W1: Different model or weights will require retraining of proposed projector, making the method less portable and generalizable. It also require white-box model, making it unusable for blackbox API which is the most powerful and suitable model for high stack scenarios."

We thank the reviewer for raising this question. We would like to emphasize that the primary goal of this work is to investigate the internal mechanism of the G–D Gap and develop targeted corrections using the VLM's own representations. As described in Section 3.3, BRIDGE extracts reliability-related signals from hidden states HH through a lightweight Projector, producing a risk score s=fθ(H)s=f_{\theta}(H). This score determines the strength of verification intervention during generation. White-box access is therefore central to our mechanism analysis, allowing us to directly study and utilize the target VLM's internal information.

For black-box API models, although hidden states and internal injection positions are inaccessible, the core idea of BRIDGE can be adapted through external detection and selective re-prompting. Specifically, we can train an independent visual reliability detector on reliable and unreliable images to estimate the risk score s=fθ∗(I)s=f_{\theta}^{*}(I). Based on this score, the system can selectively add discrimination instructions and request the black-box VLM to verify or regenerate its response.

This approach preserves BRIDGE's risk-guided selective correction without requiring internal model access. However, external visual features may be less aligned with the target VLM's internal reliability signals. Therefore, our focus remains on understanding the G–D Gap under white-box access, while external detection combined with re-prompting offers a feasible extension to black-box APIs that warrants further evaluation.

AuthorsOpenReview ↗

Response 2

Comment 2:

"W2: Benefit of BRIDGE’s routing design is insufficiently isolated...The existing ablations do not replace a simple probe using the same prefix features and intervention policy."

Re: We appreciate the request for a simpler probe under matched experimental conditions. To isolate the benefit of BRIDGE's router design, we replaced our router with Logistic Regression (Linear) and MLP classifiers, using the same cosine-center prefix features, data splits, and multi-level intervention policy. Both baselines were trained using only BCE loss, without our specialized losses in Section 3.3.

On VLind, Linear and MLP achieve higher detection PR-AUC (0.988 and 0.984) than BRIDGE (0.979). However, their performance under the same multi-level routing policy is lower:

MethodPRF1Acc
MLP (cosine)0.8040.7800.7920.795
Linear (cosine)0.8570.7800.8170.825
BRIDGE0.8670.7800.8210.830

To further examine whether a simpler single-threshold decision would suffice, we additionally evaluated binary routing with these classifiers:

MethodPRF1Acc
BRIDGE0.8670.7800.8210.830
Linear (single threshold)0.7960.8600.8270.820
MLP (single threshold)0.7210.8800.7930.770

Linear (single threshold) achieves slightly higher F1 (0.827 vs. 0.821), but its precision decreases from 0.867 to 0.796, and accuracy drops from 0.830 to 0.820. MLP (single threshold) exhibits a larger precision decrease (0.721) and lower F1 (0.793). Thus, single-threshold routing favors recall, whereas BRIDGE achieves higher precision and accuracy.

We also observed that Linear and MLP scores concentrate within narrow ranges, increasing sensitivity to threshold selection. We will include the corresponding score distributions in the revised manuscript.

Together, these experiments show that higher binary separability does not guarantee better routing performance. While a single-threshold Linear router remains competitive in F1, BRIDGE offers a more favorable precision–accuracy trade-off. The controlled multi-level comparisons further support the effectiveness of our router design and intervention policy.

AuthorsOpenReview ↗

Response 3

Comment 3:

"W3: BRIDGE induces substantial inference cost, and can up to 2x higher latency. The paper reports this overhead, but lacks an end-to-end reliability/cost comparison with simpler prompting and routing alternatives."

Re: We appreciate the reviewer’s attention to the practical cost of BRIDGE. We acknowledge that BRIDGE introduces additional inference overhead. However, this cost is justified by the reliability improvements and the prevention of potentially more costly downstream errors. Moreover, selective routing avoids unnecessary interventions on reliable inputs, further improving the cost–reliability trade-off.

1. Intervention frequency. We report injection rates across four datasets:

DatasetPositive Injection RateNegative Injection Rate
VLind96.0%22.0%
IMD202095.0%77.0%
Columbia95.0%25.0%
WHOOPS87.0%—

WHOOPS contains only positive test samples.

BRIDGE intervenes on 96% and 95% of unreliable inputs in VLind and Columbia, while avoiding intervention on 78% and 75% of reliable inputs, respectively. IMD2020 has a higher negative injection rate (77%) because its anomalies are more subtle in both semantic inconsistencies and splicing artifacts (Appendix B, Table 6), requiring more conservative verification.

2. Inference latency. We additionally measured inference time per sample:

DatasetNone (s)Low (s)Medium (s)High (s)
VLind61.892.691.192.5
IMD202031.258.449.455.4
Columbia34.248.852.045.7
WHOOPS40.766.669.966.4

The maximum observed latency increase is 1.87× (IMD2020, Low). VLind was evaluated on an RTX 4080, while other datasets used an RTX 4090. Latency does not consistently increase with intervention strength because routing and attention-sink operations are shared across levels, while generation time varies.

Importantly, as illustrated in Figure 1 and Section 4.3, unreliable outputs can cause more costly downstream errors, including incorrect recommendations and contaminated records. Thus, although BRIDGE incurs additional computation, its selective verification is intended to balance inference cost against the risk of unreliable information propagation.

As discussed above, BRIDGE introduces additional inference overhead to prevent potentially more costly downstream errors. Nevertheless, we also considered a simpler, lower-cost alternative: external routing, as discussed in Section 4.7.

BRIDGE's overhead mainly comes from hidden-state detection, attention-sink computation, and additional generation. External routing reduces this cost by using an independent image classifier to select verification prompts before VLM inference. We compare their performance in Figure 5:

MethodIMD2020 F1VLind F1Columbia F1WHOOPS Recall
CNN Router0.6840.8310.8350.670
ResNet Router0.7000.8360.7230.800
ViT Router0.7130.8070.7760.750
BRIDGE0.7580.8210.8780.630

We additionally measured inference latency:

MethodBRIDGECNN RouterResNet RouterViT Router
Time (s/sample)84.536.336.738.8

External routing significantly reduces inference time. CNN and ViT achieve 57.0% and 54.1% lower latency than BRIDGE, respectively. ResNet also achieves competitive performance, leading on VLind and WHOOPS. However, its F1 drops to 0.723 on Columbia, compared with 0.878 for BRIDGE. BRIDGE also performs best on IMD2020.

These results demonstrate an efficiency–reliability trade-off. External routing is a practical lower-cost alternative, while BRIDGE offers stronger performance on IMD2020 and Columbia by leveraging the target VLM's internal representations. More importantly, our primary objective is to investigate the internal G–D Gap mechanism, rather than exclusively minimize inference latency.

Reviewer_g89iOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the "Generation–Discrimination (G-D) Gap" in VLMs: under unreliable visual inputs, models recognize unreliability when explicitly prompted but fail to express it during descriptive generation. The authors propose BRIDGE, a lightweight directional projector that scores input unreliability from prefix activations and routes to one of four intervention levels (none/low/medium/high), injecting verification prompts at attention-sink punctuation positions. On Qwen3-VL, BRIDGE improves F1 over descriptive prompting on IMD2020 (0.400→0.758), Columbia (0.033→0.878), and VLind (0.530→0.821), with downstream evaluations showing reduced unreliable propagation.

Critical strengths
  1. This work is well motivated. The G-D Gap is a precise, underexplored question. The three-signal analysis (linear-probe separability, visual attention ratio, top-k JS divergence) supports a coherent "expressive, not perceptual" narrative.

  2. This paper proposes a lightweight method. The directional projector with antipodal + Fisher losses is a sensible scalar score from frozen representations. Multi-level thresholding avoids binary brittleness, and attention-sink prompt injection is a clever low-overhead strategy that does not perturb internal states or logits.

  3. Downstream evaluation shows practical relevance. The 600-sample study across documentation, recommendation, and structured entry tasks is a meaningful step beyond benchmark F1; UPR alongside TSR credibly shows reduced propagation.

Critical weaknesses
  1. The paper does not show whether projector scores correlate with token-level JS divergence, nor whether injection increases JS divergence between reliable/unreliable outputs. A direct link would substantially strengthen the argument.

  2. The "Qwen+System Prompt" baseline conflates routing with prompt content. A stronger control is the same routing mechanism with the same verification prompts but binary (not multi-level) granularity—isolating whether gains come from routing or hierarchical design. PrivGate is adapted with a fixed reasoning prompt, a weaker intervention than BRIDGE's multi-level prompts, so the comparison is not apples-to-apples for the mechanism.

  3. The linear probe shows separability, but performance of a simple logistic regression on raw hidden states (without cosine-center aggregation) is unreported. If the aggregation is critical, it should be shown; if not, its added complexity needs justification. Similarly, the attention analysis lacks a suppressed-attention control to establish whether the pattern is diagnostic or incidental.

Questions for authors
  1. What is performance with a single fixed threshold set across all four Qwen datasets? If degradation is substantial, what does this imply for generalization to new unreliability types?

  2. What proportion of samples receive each intervention level per dataset, and how does this correlate with the PPL/latency increases?

  3. Does BRIDGE injection increase JS divergence between reliable and unreliable outputs at the injection position and subsequent tokens?

  4. What is performance of a binary router (score > single threshold) using the same high-level verification prompt—isolating multi-level granularity from routing itself?

8 comments
AuthorsOpenReview ↗

Response 1

Comment 1:

"W1: The paper does not show whether projector scores correlate with token-level JS divergence, nor whether injection increases JS divergence between reliable/unreliable outputs. A direct link would substantially strengthen the argument."

Re: We thank the reviewer for this important question. To examine the relationship between Projector Score and token-level JS divergence, we conducted two complementary experiments on 200 VLind samples from 100 paired scenes.

1. Correlation between Projector Score and token-level JS divergence.

For Ordinary and BRIDGE, we calculate the JS divergence between each sample's next-token distribution and the mean distribution of reliable samples from other scenes under the same condition. Spearman correlations are computed using the same Projector Score.

Across the full generation sequence, both conditions yield ρ=−0.099\rho=-0.099, with 95% CIs of [−0.228,0.036][-0.228,0.036] for Ordinary and [−0.235,0.035][-0.235,0.035] for BRIDGE. Excluding the first 23.8% of generation gives −0.106-0.106 [−0.240,0.030][-0.240,0.030] and −0.061-0.061 [−0.200,0.078][-0.200,0.078], respectively. Thus, no clear global monotonic correlation is observed.

Position-wise correlations vary under BRIDGE, from −0.450-0.450 [−0.565,−0.335][-0.565,-0.335] at 26.2% of generation to 0.3300.330 [0.213,0.442][0.213,0.442] at 97.6%. The corresponding Ordinary values are −0.139-0.139 and 0.0590.059, with CIs including zero. This suggests possible stage dependence, although these exploratory comparisons are not corrected for multiple testing.

2. Effect of verification intervention on output-level class separation.

More importantly, we directly examine whether BRIDGE strengthens reliability-related separation in output distributions. We define the Separation Gap as

\frac{ JS_{\mathrm{within},0}(t)+JS_{\mathrm{within},1}(t) }{2},$$ where the terms measure between-class and within-class JS divergence. A larger gap indicates stronger class separation relative to within-class variation. Using the same 200 samples, each JS comparison uses two disjoint groups of 50 samples, with subgroup-mean distributions computed before JS divergence. We repeat sampling 50 times and analyze the complete **151,936-token vocabulary** across **21 generation intervals**. Statistical significance is assessed using **499 within-scene label permutations**, preserving scene pairing and class sizes. | Metric | Ordinary | BRIDGE | |:---|:---|:---| | Observed Mean Gap | 0.00644 | **0.05963** | | Permutation Null Mean | -0.00423 | -0.00262 | | Upper-tail p-value | **0.002** | **0.002** | | Two-sided p-value | 0.004 | 0.004 | | Lower-tail p-value | 1.000 | 1.000 | Both conditions show label-associated separation relative to their permutation null distributions (upper-tail $p=0.002$). BRIDGE increases the observed mean Separation Gap from **0.00644 to 0.05963 (9.26×)**. Its between-class JS also exceeds the within-class JS of Class 0 and Class 1 by **0.04603** and **0.07323**, respectively. Overall, the first experiment reveals weak global correlations between Projector Score and token-level JS (**$\rho=-0.099$** for both conditions), with possible generation-stage dependence. **The second experiment provides the primary evidence for our mechanism hypothesis:** BRIDGE increases the mean Separation Gap from **0.00644 to 0.05963 (9.26×)**, with label-associated separation confirmed by permutation tests (upper-tail **$p=0.002$**). These results support our interpretation of the **G–D Gap as a discrepancy between internally decodable discriminative information and its expression during generation**, which BRIDGE helps alleviate. **Comment 2:** > "Q3: Does BRIDGE injection increase JS divergence between reliable and unreliable outputs at the injection position and subsequent tokens?" **Re:** We thank the reviewer for this question. As shown in Comment 1, **BRIDGE increases the mean JS Separation Gap from 0.00644 to 0.05963 (9.26×)**, indicating stronger separation between reliable and unreliable outputs relative to within-class variation. This supports the effectiveness of verification injection in enhancing output-level discrimination. The analysis covers the full generation sequence; changes in the raw between-class JS divergence at the exact injection position and individual subsequent tokens were not separately measured.
AuthorsOpenReview ↗

Response 2

Comment 3:

"W2: ...A stronger control is the same routing mechanism with the same verification prompts but binary (not multi-level) granularity—isolating whether gains come from routing or hierarchical design..."

Re: We appreciate the reviewer’s emphasis on isolating routing granularity. To isolate the effect of multi-level routing, we conducted a controlled binary-routing experiment using the same trained classifier as BRIDGE, without retraining or modification. We evaluated all three levels' prompts for a comprehensive comparison. Each binary router uses a single F1-optimized threshold to select between no intervention and a fixed prompt.

As shown in the results above, Binary_high achieves an F1 of 0.794 and accuracy of 0.800, compared with 0.821 and 0.830 for BRIDGE. Binary_low and Binary_medium achieve F1 scores of 0.753 and 0.802, respectively. Thus, BRIDGE outperforms all three binary variants in F1 and accuracy, providing controlled evidence that multi-level routing offers benefits beyond binary routing alone. We also clarify that binary routing was already considered in our original experimental design. Specifically, we included PrivGate [1], which uses a simple linear-layer router to make a binary decision between intervention and no intervention. Its results are reported in Table 1 and reproduced below.

MethodIMD2020 F1IMD2020 AccColumbia F1Columbia AccWHOOPS RecallWHOOPS FNRVLind F1VLind Acc
PrivGate0.5880.6850.5880.7080.1600.8400.5680.665
Qwen (Ours)0.7580.7450.8780.8750.6300.3700.8210.830
Improvement+28.9%+8.8%+49.3%+23.5%+293.8%−56.0%+44.6%+24.8%

BRIDGE consistently outperforms PrivGate across all four datasets. However, since the two methods also differ in implementation, we further conducted a strict controlled binary-routing comparison to isolate the effect of routing design.

We use exactly the same trained classifier as BRIDGE, without retraining or modification. Binary routing uses a single F1-optimized threshold to select between no intervention and one fixed intervention prompt. We evaluate all three prompt strengths separately on VLind.

MethodPRF1Acc
Binary_low0.8590.6700.7530.780
Binary_medium0.7590.8500.8020.790
Binary_high0.8190.7700.7940.800
BRIDGE0.8670.7800.8210.830

BRIDGE achieves the highest F1 (0.821) and accuracy (0.830), outperforming all three binary-routing variants. The fixed prompts also show distinct precision–recall trade-offs: Binary_low has lower recall (0.670), while Binary_medium achieves higher recall (0.850) at the cost of precision (0.759).

These findings directly support Challenge 3 in our Introduction: VLMs are sensitive to prompt design, and a single fixed intervention may lead to missed detections or excessive judgments. Our multi-level routing addresses this limitation by adapting intervention strength to the detected reliability risk.

The results are also consistent with Figure 4a (Section 4.4.1), where different fixed intervention strengths produce distinct effects. Together, the original PrivGate comparison and the new controlled experiments provide stronger evidence for the effectiveness of multi-level routing over a single fixed intervention.

[1] Runshan Hu,Yukun Dong,Yingying Huangfu,Ruohan Zhao,Yi Xie, and Tieyan Li. PrivGate: Steering contextual integrity in LLMs via latent spacegeometry. In Proceedings of the 43rd International Conference on Machine Learning, 2026. URL: https://openreview.net/forum?id=oKizUOP7E4

Comment 4:

"Q4: What is performance of a binary router (score > single threshold) using the same high-level verification prompt—isolating multi-level granularity from routing itself?"

Re: We appreciate the reviewer’s specific request for a high-level binary-routing control. We kept BRIDGE’s trained classifier unchanged and used a single F1-optimized threshold to choose between no intervention and one fixed prompt. Beyond the requested High prompt, we also tested Low and Medium.

On VLind, Binary_high obtains F1 = 0.794 and accuracy = 0.800, versus 0.821 and 0.830 for BRIDGE. Binary_low and Binary_medium obtain F1 scores of 0.753 and 0.802. Thus, BRIDGE achieves higher F1 and accuracy than all three binary controls, supporting the value of multi-level routing with the detector held fixed.

AuthorsOpenReview ↗

Response 3

Comment 5:

"W3: ...but performance of a simple logistic regression on raw hidden states (without cosine-center aggregation) is unreported. If the aggregation is critical, it should be shown; if not, its added complexity needs justification... the attention analysis lacks a suppressed-attention control to establish whether the pattern is diagnostic or incidental."

Re: We thank the reviewer for these questions. The importance of aggregation was already evaluated in Section 4.4.2 (Figure 4b). Here, single token denotes classification using a raw hidden state without cosine-center aggregation. The IMD2020 results are:

MethodToken 1020304050
Single token0.9150.9460.9170.9020.873
Detector (cosine center)0.9640.9720.9710.9720.961
Relative decrease−5.10%−2.69%−5.54%−7.17%−9.21%

Without aggregation, performance decreases by 5.94% on average, supporting the cosine-center design in Section 3.1.

Comparison with simpler classifiers. We additionally trained Logistic Regression (Linear, 4,097 parameters) and MLP (2,106,369 parameters) on raw hidden states and cosine-center representations. Linear uses a 4096→1 layer; MLP uses LayerNorm, 4096→512 projection, GELU, dropout (0.1), and an output layer. Both use only BCE loss, without our Section 3.3 losses. All experiments use the same VLind data splits.

PR-AUCBRIDGEMLP (cosine)Linear (cosine)MLP (raw)Linear (raw)
Score0.9790.9840.9880.9460.911

Cosine-center aggregation improves both classifiers. Although Linear and MLP achieve higher PR-AUC than BRIDGE, binary separability is not the sole objective of our router. We therefore evaluated their effectiveness within the same multi-level routing framework:

MethodPrecisionRecallF1Accuracy
MLP (cosine)0.8040.7800.7920.795
Linear (cosine)0.8570.7800.8170.825
BRIDGE0.8670.7800.8210.830

BRIDGE achieves the best F1 (0.821) and accuracy (0.830). We also observed that Linear and MLP scores concentrate within narrow ranges, increasing sensitivity to routing thresholds. We will provide these score distributions in the revised manuscript. These findings support the effectiveness of our router design for multi-level intervention, beyond binary classification performance.

Attention suppression experiment. We further tested the role of visual attention (Section 3.2) using the discrimination prompt on VLind. Image-key attention weights are scaled by α\alpha and renormalized:

aj′=ajwj1−mV+αmVa'_j = \frac{a_j w_j}{1-m_V+\alpha m_V}

where wj=αw_j=\alpha for image keys (j∈Vj\in V) and wj=1w_j=1 otherwise, with mV=∑j∈Vajm_V=\sum_{j\in V}a_j. Here, VV denotes image keys, and mVm_V is their original attention mass. The resulting mass is αmV/(1−mV+αmV)\alpha m_V/(1-m_V+\alpha m_V), so the actual reduction depends on the initial attention distribution.

ConditionPhaseImage Attention MassPrecisionRecallF1Accuracy
BaselineDecode0.2130.5321.0000.6940.560
BaselinePrefill0.1520.5321.0000.6940.560
α = 0.1Decode0.0720.5321.0000.6940.560
α = 0.1Prefill0.0540.5321.0000.6940.560
α = 0.5Decode0.1610.5280.9500.6790.550
α = 0.5Prefill0.1130.5280.9500.6790.550

Although α = 0.1 substantially reduces image attention mass, performance remains unchanged. At α = 0.5, F1 decreases from 0.694 to 0.679. Thus, visual attention mass is not directly proportional to discrimination performance.

Overall, the results show that cosine-center aggregation improves reliability detection, while the proposed router better converts detection scores into effective interventions. The attention results further distinguish access to visual information from its effective use during generation, consistent with our discussion of the G–D Gap.

Reviewer_HPEEOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the generation–discrimination gap in vision–language models (VLMs): models often fail to flag unreliable visual content during descriptive generation despite recognizing it under explicit verification. Analyses of hidden representations, visual attention, and output distributions suggest that reliability-related information remains internally accessible but is weakly expressed in generated responses. The authors propose BRIDGE, which uses a lightweight projector to estimate unreliability from intermediate representations and selectively injects verification prompts of varying strengths at attention-sink positions. The VLM remains frozen throughout. Across four benchmarks covering image manipulation and semantic unreliability, BRIDGE outperforms the evaluated VLM-based baselines on the primary recognition metrics. In three downstream tasks, it reduces unreliable information propagation by 11.7 percentage points while slightly improving task success, at the cost of increased inference latency.

Critical strengths

The method provides a flexible mechanism for selective verification. BRIDGE uses an input-specific risk score to determine whether verification is needed and how strongly to intervene. This design addresses the balance between detecting unreliable inputs and avoiding unnecessary warnings on reliable ones. The intervention ablations provide empirical support for this adaptive strategy.

The framework makes practical use of a frozen VLM’s representations. BRIDGE trains a small auxiliary projector and uses textual prompts to guide correction, without updating the backbone or directly editing its activations or logits. This modular design provides a way to add reliability-aware generation to existing VLMs, and its applicability is demonstrated on three backbones in the IMD2020 experiments.

Critical weaknesses

The interpretation is stronger than the evidence supports. Section 3.2 and the conclusion attribute the G–D gap primarily to insufficient expression of internally available discriminative evidence. However, linear separability of hidden representations, substantial visual attention, and weak output-level separation do not establish this explanation as the primary mechanism. In particular, decodable information does not necessarily imply that the model has already formed a correct reliability judgment, and explicit verification may elicit additional reasoning.

The evaluation does not systematically assess the correctness of verification explanations. The main evaluation maps generated responses to binary recognition labels and reports metrics such as F1 and Recall. These metrics cannot distinguish a correct reliability judgment supported by valid evidence from a correct label accompanied by an inaccurate or unsupported explanation. This leaves an important aspect of generation reliability insufficiently evaluated.

Questions for authors

N/A

5 comments
AuthorsOpenReview ↗

Response 1

Comment 1:

"W1: The interpretation is stronger than the evidence supports... However, linear separability of hidden representations, substantial visual attention, and weak output-level separation do not establish this explanation as the primary mechanism..."

Re: We appreciate the reviewer's concern about the strength of our mechanism interpretation. We would like to clarify that the Generation–Discrimination Gap (G–D Gap) describes the discrepancy between a VLM's ability to generate correct content under unreliable visual inputs and its ability to evaluate their correctness, rather than simply detecting unreliable inputs or assessing output quality. We quantify it as

ΔG−D=M(y,y^D)−M(y,y^G),\Delta_{\mathrm{G-D}}=\mathcal{M}(y,\hat y_D)-\mathcal{M}(y,\hat y_G),

where M\mathcal{M} denotes Recall or F1. In Figure 3a, discrimination prompts improve F1 from 0.400 to 0.778 on IMD2020 and 0.530 to 0.694 on VLind, demonstrating this gap.

Section 3.2 investigates three stages of generation. Figure 3b reveals decodable discriminative information in intermediate representations, while Figure 3c shows continued visual attention. However, Figure 3d indicates weak output-level class separation. These observations suggest a bottleneck in expressing internal information through final output probabilities:

pt=O(HL,t)=softmax⁡(WLMHL,t+bLM).p_t=O(H_{L,t})=\operatorname{softmax}(W_{\mathrm{LM}}H_{L,t}+b_{\mathrm{LM}}).

To further examine this interpretation, we conducted additional experiments on 200 VLind samples from 100 paired scenes, comparing representations, attention, and output distributions before and after BRIDGE intervention.

First, we examined the separability in hidden representations HlH_l. If insufficient internal separability were the primary bottleneck, improved generation would be expected to accompany greater separability. However, SVM F1 remains largely unchanged after BRIDGE, despite improved generation performance:

LayerSVM F1 BeforeSVM F1 BRIDGEChange
00.9030.923+0.020
60.9000.912+0.012
120.9200.919−0.001
180.9420.939−0.003
240.9440.937−0.007
300.9320.917−0.015

This suggests that BRIDGE improves the expression of existing discriminative information, rather than substantially increasing its separability.

Second, we examined the visual attention. Since attention influences hidden-state formation, substantial information loss through attention would be expected to affect downstream separability. We therefore measured Image Attention Mass:

LayerBeforeBRIDGEChange
00.0640.019−0.045
60.0380.015−0.023
120.1080.040−0.068
180.3210.126−0.195
240.2220.096−0.126
300.0710.030−0.041

Despite reduced visual attention, the downstream hidden representations retain comparable separability. Together with Figure 3c, this is consistent with verification-oriented attention redistribution, potentially emphasizing anomaly-related evidence and textual instructions. The results therefore suggest that output-level expression, rather than substantial loss of decodable information, is a key factor.

Third, we examined output-level separation. We analyzed the full 151,936-token vocabulary across 21 generation intervals, with 50 balanced JS comparisons. We define the Separation Gap as

\frac{ JS_{\mathrm{within},0}(t)+JS_{\mathrm{within},1}(t) }{2},$$ | Metric | Ordinary | BRIDGE | |:---|:---|:---| | Observed Mean Gap | 0.00644 | **0.05963** | | Permutation Null Mean | −0.00423 | −0.00262 | | Upper-tail p-value | 0.002 | 0.002 | | Two-sided p-value | 0.004 | 0.004 | | Lower-tail p-value | 1.000 | 1.000 | The mean Separation Gap increases from **0.00644 to 0.05963**, indicating more pronounced output-level class separation following BRIDGE intervention. Both conditions show label-associated separation under **499 within-scene permutations** (upper-tail $p=0.002$ each). Taken together, these results support **insufficient expression of internally decodable discriminative information at the final generation stage** as a key explanation for the G–D Gap. Importantly, **BRIDGE aims to preserve and better express existing discriminative information, rather than strengthen the VLM's intrinsic discriminative capability**, thereby narrowing the gap between Generation and Discrimination.
AuthorsOpenReview ↗

Response 2

Comment 2:

"W2: The evaluation does not systematically assess the correctness of verification explanations. The main evaluation maps generated responses to binary recognition labels... These metrics cannot distinguish a correct reliability judgment... This leaves an important aspect of generation reliability insufficiently evaluated."

Re: We thank the reviewer for emphasizing the correctness of verification explanations. We would like to clarify that this concern was considered in our experimental design, as reflected in Section 4.3 (Table 2). VLM-generated descriptions serve as inputs to three downstream tasks: Reference Documentation, Recommendation Generation, and Structured Information Entry. All methods use identical downstream settings. Incorrect descriptions or unsupported explanations may therefore reduce task success (TSR) or increase unreliable information propagation (UPR).

Table 2 reports the following results:

MethodDoc. TSRDoc. UPRDoc. F1Rec. TSRRec. UPRRec. F1Struc. TSRStruc. UPRStruc. F1
Descriptive Prompt90.5%13.0%83.7%91.0%14.0%77.0%63.0%54.0%49.2%
BRIDGE95.0%2.0%87.9%93.5%4.0%81.8%64.0%40.0%52.0%

Doc. refers to Documantation, Rec. refers to Recommendation Generation, Struc. refers to Structured Information Entry

BRIDGE achieves higher TSR and F1 in all three tasks. It also reduces UPR from 13.0% to 2.0%, 14.0% to 4.0%, and 54.0% to 40.0%. These results support the reliability of BRIDGE's corrections in downstream applications.

To directly assess explanation correctness, we additionally evaluated 60 Columbia and 100 WHOOPS positive samples using human annotators and GPT-6 Sol. Both followed the same five-point rubric, covering judgment accuracy, visual grounding, and unsupported claims.

JudgeDatasetNReliability RateAvg. Score1 (Incorrect)2 (Unsupported)3 (Vague)4 (Mostly Grounded)5 (Well Grounded)
HumanColumbia6092.59%3.8764151916
HumanWHOOPS10089.06%4.3036761239
GPT-6Columbia6077.78%3.4161214226
GPT-6WHOOPS10089.06%3.7536711379

Reliability rates and average scores refer to successful corrections; score distributions cover all evaluated samples.

Among successful corrections, human-rated explanation reliability reaches 92.59% on Columbia and 89.06% on WHOOPS, with average scores of 3.87 and 4.30. GPT-6 Sol reports corresponding reliability rates of 77.78% and 89.06%, supporting the visual grounding of corrected explanations.

Together, the downstream evaluation and direct annotation provide complementary evidence that BRIDGE improves explanation reliability and reduces the propagation of unreliable information, beyond binary recognition alone.

AuthorsOpenReview ↗

Response 3

We sincerely appreciate your insightful comments on the interpretation of the G–D Gap and the reliability of generated explanations. Your feedback has helped us refine our mechanistic claims and strengthen the supporting evidence. We have incorporated these revisions into the updated manuscript, with the relevant details located below:

  • G–D Gap mechanism and post-intervention analysis: Section 4.4; Appendix E.
  • Verification explanation quality and downstream reliability: Section 4.3; Appendix G.

We hope these additions provide a clearer and more rigorous account of the proposed mechanism and the reliability of BRIDGE's outputs. We would welcome any further questions or suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.