acceptodds
Under review as a conference paper at ICLR 2027

Correspondence, Not Similarity: Gating Multi-View Evidence in Driving Vision-Language Models

Abstract

Driving vision-language models are given several cameras, and a growing number of evaluations conclude that a model used the extra view. The usual evidence is a matched negative: show the model a comparable image containing no target object, and check that its answer returns to baseline. We show that this control certifies less than it is taken to certify. Holding scene, timestamp, ego pose and ground truth fixed across 610 Paired Exposure Units built from nuScenes, we replace the true side camera with one from another vehicle's world, matched on city, hour, weather and road-user status. Across five vision-language models the true and the substituted view prove equally persuasive, and two models find the substituted view more persuasive. The distinction is easy to draw from pixels, where an off-the-shelf matcher separates the two at 0.973 AUROC. We benchmark ten verification methods and find that those measuring correspondence keep the evaluation valid at every difficulty, while those measuring similarity fail regardless of supervision. Gating on correspondence needs no extra generation. We release the benchmark, the pools, and all 122,000 responses.

Primary area:
datasets and benchmarks
Keywords:
vision-language models, evaluation methodology, multi-view evidence, benchmark, autonomous driving

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1mqZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that the standard control used to certify that a multi-camera driving VLM "used" an additional view, the matched negative (an image of the same scene without the target object), only tests whether the added image carries the target content, and not whether it comes from the scene the question is about (provenance). To separate the two, the authors build a benchmark of 610 Paired Exposure Units on nuScenes, in which the vehicle's true synchronized side view is replaced by a side view taken by another vehicle at the same camera position ("wrong world"), drawn from four donor pools of increasing similarity (RANDOM, CITY, CONTEXT, HARD).

The proposed measure, CSCE, is the difference between the effects of the true and the substituted view on the commitment rate. Across five open-weight VLMs, CSCE is not positive in 13 of 20 model×difficulty cells. The paper then compares ten pair-verification methods used as a gate in front of the VLM, and finds that correspondence-based methods (SIFT+RANSAC, LoFTR, and the proposed training-free RPC score) restore a positive CSCE in all cells, while global-embedding methods do not.

Critical strengths
  1. Careful experimental controls. Splits are built over dependency links; the pair used to mine donors is kept disjoint from the pair scored by the gate, with a full mining×scoring matrix; hyperparameters are fixed on the easiest pool only; and cross-dataset transfer is done without refitting. Within the problem setting the paper defines, the comparisons between verification methods are credible, and the graded donor pools show clearly that global-similarity methods break down on hard pairs while local matching remains effective.
  2. Strong reproducibility. The supplementary material includes code and number-checking scripts, and the authors commit to releasing all model responses and verification scores, which makes the benchmark easy to build on.
Critical weaknesses
  1. The motivation is detached from real deployment, and the paper gives no corresponding real-world scenario. In an actual vehicle, the source of every camera stream is fixed: the identity of the front and side cameras and their time synchronization are guaranteed by the system. The constructed setting, in which another vehicle's side view is fed to the model, does not occur in normal operation. Under this premise, what the matched negative tests (whether the model responds to the content of the image) is exactly what a real system needs; the provenance check the paper asks for only matters when inputs are corrupted or tampered with. If the authors consider the problem practically relevant, they should study failure modes that actually occur, such as camera time desynchronization, calibration errors, cameras mis-assigned in the data pipeline, or adversarial image injection. None of these is addressed. Moreover, the paper positions itself as a study of evaluation protocols, yet the Ethics statement invokes driving safety to argue for the importance of the problem; the two framings are inconsistent.
  2. The generality of the main finding is questionable. The prompt used in the main experiments does not state which camera each image comes from, nor that the images are synchronized (the authors note this explicitly in prompts.py in the supplementary material). Only open-weight general-purpose VLMs up to 32B parameters are tested. The paper does not test a setting in which camera roles and synchronization are stated explicitly and the model is asked to check the overlapping region before answering, nor does it test stronger models. The conclusion that VLMs do not verify view provenance may therefore hold only for small models with a minimal prompt.

Why these determine my recommendation. In real systems the source of each camera is known, and the content-level evidence use tested by the matched negative already meets practical needs; the provenance problem only arises when inputs are faulty, and no real form of input failure is studied (W1). Even within the paper's setting, the central observation that models do not verify provenance is supported only by small open-weight models and a prompt that gives no camera information (W2). Together, these leave the significance and the generality of the contribution below the bar for acceptance at ICLR, despite the careful execution.

Questions for authors

Questions that could change my assessment

  1. In a deployed system, what scenario corresponds to "receiving another vehicle's view"? Can the conclusions be validated under real input failures (camera time desynchronization, calibration errors, mis-assigned cameras, adversarial injection)?
  2. Which published claims of multi-view evidence use would be invalidated by this work? Please list them concretely.
  3. If the prompt explicitly states the camera role of each image and their synchronization, and asks the model to compare the overlapping region and reason explicitly before answering, do the results change? Do they still hold for stronger models, including proprietary ones?
  4. Why is the provenance control experiment in the supplementary log (neutral / verify / disclosed) not included in the main paper? Is the headline result (CSCE ≤ 0 in 13 of 20 cells) stable across prompt wordings?

Suggestion (does not affect my assessment)

  • A glossary defining PEU, CSCE, WLR and the four donor pools in one place would help readers.
Reviewer_GmuKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether driving VLMs can distinguish camera views from the same scene from views taken elsewhere. The authors argue that existing matched-negative evaluations only test whether a model uses the content of a second view, without checking whether that view belongs to the current scene. They build a benchmark of 610 units from nuScenes, replacing real side-camera views with images from other scenes at four difficulty levels. They find that five VLMs are similarly influenced by real and incorrect views. They then compare ten verification methods and show that local feature correspondence works better than global image similarity. The authors also propose RPC, a training-free method for filtering incorrect views.

Critical strengths
  1. The paper makes an interesting distinction between using visual content and verifying whether that content belongs to the current scene.

  2. The authors test four difficulty levels, five VLMs, and ten verification methods. The careful data splitting and additional experiments on Argoverse 2 make the evaluation more convincing.

  3. The paper provides useful analysis of the matched features and releases the model responses and verification scores.

Critical weaknesses
  1. Matched negatives are mainly used to test whether a model relies on the content of a second image. The authors additionally expect the model to verify whether that image belongs to the current scene. I think these are two different tasks. In real multi-camera systems, camera sources are usually handled through calibration and synchronization. Therefore, a model trusting an incorrect input does not necessarily mean it failed to use visual evidence correctly. The paper shows that VLMs do not actively verify provenance, but I am not fully convinced that this makes existing matched-negative evaluations unreliable. Also, the proposed gate checks visual correspondence, but may not detect outdated frames from the same scene. This limits what the method can actually verify.

  2. The paper emphasizes that gating makes CSCE positive in all 20 settings. However, some confidence intervals still include zero. Moreover, rejected views are replaced with front-only responses by design. The improvement therefore mainly reflects the gate's ability to filter views, rather than any improvement in the VLM's own ability to verify evidence.

  3. The main experimental finding is that local feature matching works better than global image similarity. However, this is already a well-established idea in image matching and visual place recognition. RPC is simple and efficient, but its performance is below existing methods. On HARD, its AUROC is 0.836, compared with 0.869 for SIFT+RANSAC and 0.973 for LoFTR. It also admits many incorrect views on Argoverse 2 when using the original threshold. The results provide stronger evidence for correspondence-based verification in general than for RPC as a new method.

Questions for authors
  1. Why should provenance verification be handled by the VLM itself rather than by the camera system? How does this limitation affect the conclusions drawn from existing matched-negative evaluations?

  2. Apart from computational efficiency, what is the main advantage of RPC over existing methods such as LoFTR?

Reviewer_hSGEOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks what the standard matched-negative control in multi-view driving VQA actually certifies. Removing the target object from the added camera image shows that the model responded to content. It does not show that the image comes from the scene the question asks about. The authors name the second property provenance and build 610 Paired Exposure Units from nuScenes keyframes, replacing the synchronized side view with one from a different vehicle in the same city, hour and weather, while scene, timestamp, ego pose and ground truth stay fixed. Four donor pools, RANDOM through HARD, each remove one further confusable cue. Over five open-weight VLMs and 122,000 responses, the true and the substituted view shift commitment rates indistinguishably, and on HARD both Qwen3-VL models are negative at -0.230 and -0.220, so the foreign view moves the answer further than the true one.

The second half benchmarks ten methods for deciding whether two views show the same world. Supervision is not what separates them. CLIP, SigLIP2, DINOv2 and the three place recognition models fall toward a colour histogram on the matched pools, while SIFT with RANSAC, LoFTR and the authors' RPC hold their discrimination. RPC scores frozen DINOv2-base patch tokens by averaging top-K mutual nearest-neighbour similarities, reaches 0.836 AUROC on HARD at 2.1 ms per pair, and sits 13.7 points below LoFTR. Gating the second view on it returns a positive CSCE in all 20 model-by-difficulty cells, admitting 36% of foreign views on HARD. Argoverse 2, scored with the fitted thresholds and no refitting, reproduces the ordering of the method families.

Critical strengths
  • The paper breaks a licensing step the driving VLM literature leans on. A matched negative certifies that the model received the target content, not that the content came from the right scene, and the difference-in-differences contrast shows that five VLMs cannot tell the two apart over 610 units. On HARD both Qwen3-VL models are negative at -0.230 and -0.220, so the substituted view is the more persuasive one. The design separates the provenance effect from the content effect, which makes the failure attributable rather than suggestive.
  • The benchmark is built so that the conclusion does not rest on the authors' own scorer. HARD is mined front to front and never on the pair a gate scores; the mining-by-scoring matrix in Appendix A.1 holds SIFT at 0.869 to within a thousandth across all three mined pools; and the Argoverse 2 transfer in Section 5.3 reuses the fitted thresholds with no refitting. The paper also states where its own method is weakest, including the 13.7-point gap to LoFTR on HARD and false-positive rates of 0.775 and 0.431 on Argoverse 2 against 0.071 and 0.076 for SIFT.
Critical weaknesses
  • The main claim is reported without the intervals the paper says it computes. Section 5 states that intervals resample the 49 dependency components, the Limitations section calls 49 a coarse unit for a bootstrap, and Table 2 and Figure 4b then give point estimates only. The HARD column is where the claim lives and its gaps are small. SIFT reaches 0.171, LoFTR 0.196, RPC 0.100, and the supervised head -0.001. A reader cannot tell whether the three-tier ordering survives resampling, or whether "restores the measurement in every cell" is a point estimate that an interval would soften.
  • The behavioural finding rests on one question template, one dataset and five models under 32B parameters. Whether a pedestrian or cyclist could enter the ego vehicle's path within 3 s is the only question asked, the 610 units are all nuScenes keyframes, and Section 5.3 transfers the verification methods to Argoverse 2 rather than the models. Nothing in the data extends "the models do not check provenance" to another question type, to a counting or signal-state task, or to the larger proprietary models a practitioner would deploy. The paper states this limit; it still bounds what the result establishes.
Questions for authors

Which encoder mined the HARD pool behind Tables 1 and 2? The main text gives the criterion, the donor whose front view most resembles the recipient's front view, but not the model that measures the resemblance. Table 3 reports a different AUROC per method for each of the three mined pools and describes the DINOv2-mined pool as the hardest for both CLIP rows, so the choice decides whether the headline numbers sit at the most or the least favourable point for the similarity family.

Can the bootstrap intervals be reported in Table 2? Section 5 describes them over the 49 dependency components and no table or figure shows them. The three-tier reading of the gating results depends on the gap between LoFTR at 0.196 and RPC at 0.100 on HARD, and on the gap between RPC and the supervised head at -0.001, both of which an interval could close.

What is the full per-query cost of the gating policy in deployment? Offline both branches are already in the response set, so re-selection is free and the paper is right that no extra generation is needed there. Serving a live query needs the front-only answer and the two-view answer, which is a second VLM pass that the 2.1 ms per pair for RPC does not cover. The timings in Table 1 need a device named as well. The ordering between 2.1 ms and 99.4 ms can change between CPU and GPU, and the forty-times comparison against EigenPlaces in Section 5.2 is interpretable only with the hardware stated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.