- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper emphasizes that a critical point in evaluating vision-language models that the records were not being treated seriously. Therefore, it provided a benchmark for auditable verification ContraLedger, whose core design is “Counterfactual Twins”, “Executable Specs” and “Read-Know-Decide Protocol”. This paper proposed a fine-grained approach for failure analysis, as well as conducted empirical study and decomposition experiments.
Critical strengths
- I tend to think the authors are targeting to an important aspect in visual scenes and they point to a very certain task.
- The technical contributions well serve for the problem to solve (even though the results are not always positive). I also check the code package, which seems reliable.
Critical weaknesses
- This paper is weak in presentation which takes long time to follow. I strongly suggest authors to revise the writing carefully, such as unify the terms and sharpen the storyline.
- Based on the results, existing methods and ContraLedger seem lie at the same level, all show strong tendency to be misled by attacks. It seem an evidence showing ContraLedger is not good enough to prevent the attacks? Maybe you aim to prove verification failures is common and hard challenges? Then, ContraLedger cannot be regarded as a contribution, right?
- The failure results seems sensitive to response biases, and inference protocols, etc. Therefore, the visual failure still remains insufficiently isolated. It makes me a bit doubt on the broader significance.
Questions for authors
I am not sure you are actually doing exactly scene text understanding, for example, simply reproduce the known arithmetic reasoning errors in a visual-text setting (instead of raising multimodal verification)? How to tell the differences in your experimental settings?
Response on Presentation, Matched Controls, and Verification Errors
Thank you for the presentation and interpretation suggestions.
1. Presentation and contribution. We rewrote the Abstract, Introduction, contributions and Discussion, unified terminology and organized the main text into three questions about matched presentations, prerequisite controls and executable error audits. We expanded the explanations of experimental contrasts, denominators and result interpretation. Similar carrier results support a shared verification problem, without showing an attack advantage for our carrier (Table 2). ContraLedger is an evaluation protocol connecting validity twins, executable labels, rule-equivalence checks and error replay.
2. Matched image/text results. We evaluated the same 128 pairs in both media. All entries below are correct pairs /128; Qwen3 denotes Qwen3-VL-8B and Qwen3.5 denotes Qwen3.5-27B (Table 3).
Within each method, fields, public rules/assumptions, formats, decoding and stage-wise caps match. Other methods allow 256 output tokens; criterion-first allows 128+256 in both media. All 1,024 blind criteria are equivalent, yet criterion-first makes 3/256 and 23/256 completed text errors with exact fields in those same responses.
The image/text effect varies by method, and both criterion-first contrasts are non-significant after Holm correction. Text also changes representation and input length, so these results do not isolate a uniquely visual cause. Replay of 41 archived errors finds 39 arithmetic errors and two incorrect-input cases; each verdict agrees with its displayed CHECK. Figure 6a illustrates one replayable arithmetic error.
3. Bias and inference protocol. Full-set pair accuracy requires both twins correct and penalizes constant acceptance or rejection. Conditional rates can still reflect acceptance bias. Read/Know are independent calls and do not establish field use in Decide. Figure 5 and Appendix S show descriptive within-decision attention, without establishing exact reading or causal use. We distinguish completed errors, unresolved outputs and actual costs in Tables 42-43, and retain these attribution limits explicitly.