- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that the standard control used to certify that a multi-camera driving VLM "used" an additional view, the matched negative (an image of the same scene without the target object), only tests whether the added image carries the target content, and not whether it comes from the scene the question is about (provenance). To separate the two, the authors build a benchmark of 610 Paired Exposure Units on nuScenes, in which the vehicle's true synchronized side view is replaced by a side view taken by another vehicle at the same camera position ("wrong world"), drawn from four donor pools of increasing similarity (RANDOM, CITY, CONTEXT, HARD).
The proposed measure, CSCE, is the difference between the effects of the true and the substituted view on the commitment rate. Across five open-weight VLMs, CSCE is not positive in 13 of 20 model×difficulty cells. The paper then compares ten pair-verification methods used as a gate in front of the VLM, and finds that correspondence-based methods (SIFT+RANSAC, LoFTR, and the proposed training-free RPC score) restore a positive CSCE in all cells, while global-embedding methods do not.
Critical strengths
- Careful experimental controls. Splits are built over dependency links; the pair used to mine donors is kept disjoint from the pair scored by the gate, with a full mining×scoring matrix; hyperparameters are fixed on the easiest pool only; and cross-dataset transfer is done without refitting. Within the problem setting the paper defines, the comparisons between verification methods are credible, and the graded donor pools show clearly that global-similarity methods break down on hard pairs while local matching remains effective.
- Strong reproducibility. The supplementary material includes code and number-checking scripts, and the authors commit to releasing all model responses and verification scores, which makes the benchmark easy to build on.
Critical weaknesses
- The motivation is detached from real deployment, and the paper gives no corresponding real-world scenario. In an actual vehicle, the source of every camera stream is fixed: the identity of the front and side cameras and their time synchronization are guaranteed by the system. The constructed setting, in which another vehicle's side view is fed to the model, does not occur in normal operation. Under this premise, what the matched negative tests (whether the model responds to the content of the image) is exactly what a real system needs; the provenance check the paper asks for only matters when inputs are corrupted or tampered with. If the authors consider the problem practically relevant, they should study failure modes that actually occur, such as camera time desynchronization, calibration errors, cameras mis-assigned in the data pipeline, or adversarial image injection. None of these is addressed. Moreover, the paper positions itself as a study of evaluation protocols, yet the Ethics statement invokes driving safety to argue for the importance of the problem; the two framings are inconsistent.
- The generality of the main finding is questionable. The prompt used in the main experiments does not state which camera each image comes from, nor that the images are synchronized (the authors note this explicitly in
prompts.pyin the supplementary material). Only open-weight general-purpose VLMs up to 32B parameters are tested. The paper does not test a setting in which camera roles and synchronization are stated explicitly and the model is asked to check the overlapping region before answering, nor does it test stronger models. The conclusion that VLMs do not verify view provenance may therefore hold only for small models with a minimal prompt.
Why these determine my recommendation. In real systems the source of each camera is known, and the content-level evidence use tested by the matched negative already meets practical needs; the provenance problem only arises when inputs are faulty, and no real form of input failure is studied (W1). Even within the paper's setting, the central observation that models do not verify provenance is supported only by small open-weight models and a prompt that gives no camera information (W2). Together, these leave the significance and the generality of the contribution below the bar for acceptance at ICLR, despite the careful execution.
Questions for authors
Questions that could change my assessment
- In a deployed system, what scenario corresponds to "receiving another vehicle's view"? Can the conclusions be validated under real input failures (camera time desynchronization, calibration errors, mis-assigned cameras, adversarial injection)?
- Which published claims of multi-view evidence use would be invalidated by this work? Please list them concretely.
- If the prompt explicitly states the camera role of each image and their synchronization, and asks the model to compare the overlapping region and reason explicitly before answering, do the results change? Do they still hold for stronger models, including proprietary ones?
- Why is the provenance control experiment in the supplementary log (neutral / verify / disclosed) not included in the main paper? Is the headline result (CSCE ≤ 0 in 13 of 20 cells) stable across prompt wordings?
Suggestion (does not affect my assessment)
- A glossary defining PEU, CSCE, WLR and the four donor pools in one place would help readers.