- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces a detect-then-review framework designed to mitigate target-absence grounding failures in Mixture-of-Experts (MoE) Vision-Language Models (VLMs). Rather than relying on generated responses or external verifiers to detect hallucinations, the authors extract target-token routing probabilities during the standard prompt prefill to predict if a queried object is missing from the image. An L2-regularized linear detector scores these routing features, and if target absence is predicted, the system selectively triggers a target-aware review prompt to correct the premise. Evaluated on Qwen3-VL-30B-A3B-Instruct and Gemma-4-26B-A4B-it, the routing-gated policy significantly improves end-to-end accuracy on the GQA-Inpaint dataset (+22.25% for Qwen, +13.42% for Gemma) without altering model weights or router parameters.
Critical strengths
- A useful connection between detection and intervention. The framework goes beyond reporting detector AUROC by connecting a pre-generation signal to a concrete corrective action. It requires neither backbone retraining nor an auxiliary visual encoder.
- Strong In-Domain Performance. The routing probes achieve AUROCs of 0.9988 and 0.9956 on GQA-Inpaint. Scene-disjoint splits and independent processing of paired images strengthen the detector evaluation, while OBER provides an informative test of distribution shift.
- Informative controls for the selective-review policy, particularly for Qwen. Always-review and generic-review comparisons, together with results stratified by true target presence (Tables 2, 11, and 12), help distinguish the contributions of gating and prompting. The paper also measures correct-to-incorrect changes following false-positive triggers (Sec. 4.2.4, Fig. 5, and Table 13) and reports pair-clustered bootstrap confidence intervals for the selective-versus-always accuracy differences, although corresponding intervals are not provided for the headline gains over standard prompting or the detector AUROCs. Together, these analyses help readers weigh correction benefits against intervention-induced harm.
Critical weaknesses
- Routing is not shown to beat hidden states. Appendix Table 8 reports single-layer hidden-state probes reaching 0.9987 AUROC, compared with 0.9988 for the full routing representation; one also achieves higher thresholded accuracy, 98.69% versus 98.54%. These results establish that routing contains predictive information, but provide limited evidence that it is a preferable monitoring signal. A validation-selected hidden-state probe is not compared against routing on OBER or within the complete selective-review policy. Such comparisons are particularly important because routing is the paper’s central methodological distinction.
- The gate adds little beyond always-review. Selective vs. always target-aware review is 89.00% vs 86.25% on GQA (+2.75, [1.17, 4.58]) and 95.57% vs 95.30% on OBER (+0.26, [0, 0.70]), so ~11.9 of the +12.17 OBER points come from the prompt alone. On OBER the frozen threshold triggers review on 83.91% of inputs at 31.30% specificity (Table 7): the policy is nearly always-review. Gemma's always-review accuracy is not reported.
- Cost Analysis. Avoiding review calls does not by itself establish lower total inference cost. The selective policy requires a standard-prompt prefill and an additional prefill when triggered, whereas directly applying the target-aware prompt to every input could avoid the initial standard prefill. End-to-end latency or compute measurements are needed to substantiate the efficiency claim.
- Some interpretive claims exceed the supporting analyses. The abstract describes the signal as localized to the target-object token and distributed across partially substitutable experts. However, selecting target-token features does not establish localization without comparisons to other token positions. Likewise, coefficient distributions and top-k feature subsets demonstrate predictive redundancy more directly than expert substitutability. These claims should be narrowed or supported with additional controls.
Questions for authors
Please address the concerns raised in the Weaknesses section. In particular, please clarify the comparative advantages of routing over hidden-state probes, the incremental accuracy and computational benefits of selective review over always-review, and the empirical support for target-token localization and expert substitutability. I am especially interested in the practical latency implications. Although the monitor uses prefill signals, could the authors demonstrate an online implementation in which the target-absence flag is available and acted upon before any answer token is emitted? It would be helpful to report the time required to obtain the flag and compare total response latency, including review when triggered, with standard generation and direct target-aware prompting.
2 comments
Thank you for the detailed response and additional experiments. Most of my concerns have been addressed. However, the advantage of routing over hidden-state probing remains model-dependent, and selective review offers limited gains over always-review at a higher inference cost. I will therefore maintain my original rating.
Thank you for the careful review and for recognizing the value of pre-generation detection. We have added the requested comparisons and timing measurements to clarify the choice of routing, the gate's benefit, and how the signal is used online.
W1
We added the requested OBER comparison with hidden-state probes selected on GQA validation. Each policy uses its detector's fixed threshold, and comparisons are made within the same model and generation batch.
Routing performs better for Qwen, and hidden probing performs better for Gemma. To test input selection at equal coverage, we also fixed both Qwen review rates at 83.91%. Routing then reaches 95.57% accuracy and hidden gating 90.61%.
Routing also supports layer-expert analysis. The Top-500 experiment in W4 tests whether a selected feature subset retains detection performance. This makes the detector's reliance on routing features experimentally testable.
W2
Thank you for pointing out the missing Gemma always-review comparison. We evaluated all Gemma policies on the same supplementary batch of OBER answers.
These policies are close in accuracy, with always-review slightly ahead of routing-selective review. For Qwen, the following table separates the prompt's contribution from the gate's additional gain by using the same target-aware prompt.
The gate's additional benefit is clearest on GQA. On present inputs, selective review reaches 95.50% accuracy compared with 89.33% for always-review. By applying review to 50.42% of GQA inputs, the gate preserves more correct answers and achieves higher overall accuracy.
W3 / Q1
We measured both the time to obtain the flag and total response time on 200 images per model under deterministic generation. Total latency includes the original-question prefill, routing extraction, probe scoring, and answer generation.
The current implementation runs detection and generation separately and repeats prefill without KV-cache reuse. Selective review is therefore slower in this implementation. We will correct the cost claim and report the measured latency alongside the accuracy benefit described in W2.
For the online-decision part of your question, the implementation computes the routing score and selects the standard or target-aware prompt before answer generation. Runtime checks confirm that branch selection occurs before the first answer-generation forward pass completes on all 200 requests for each model. This establishes that the routing signal can actively determine the response path before an answer is emitted.
W4
Thank you for asking us to distinguish these interpretations more carefully. We compared token positions on the same 3,000 GQA test inputs, with equal feature dimensions and a separately trained probe at each position.
Detection is strong at the target token and at other question positions. We will replace “localized to the target-object token” with “detectable from routing at the target-object token.” The layer-wise probes also show that the signal is available in early layers.
For expert analysis, retraining on the top 500 routing features retains 0.9984 GQA AUC compared with 0.9988 for all features. We will replace “partially substitutable experts” with the finding that a subset of routing features retains nearly the full probe's predictive performance. These analyses establish that target absence is available in early routing signals and can be detected using a compact feature subset.