acceptodds
Under review as a conference paper at ICLR 2027

When Reading Is Not Verifying: Executable Audits Decompose Verification Failures in Vision–Language Models

Abstract

Reliable verification of text in images requires more than reading the fields or stating the right rule: a model must also execute that rule correctly. We introduce ContraLedger, an executable record-verification benchmark with matched valid/invalid twins and stage-wise diagnostics of extraction, criterion fidelity, completion, arithmetic execution, and final judgment. Across six checkpoints, 191/200 selected cases that pass source/valid controls, exact transcription, and a separate rule-rejection probe still accept the invalid record. Stronger prompting and larger test-time budgets recover many of these failures, so this 191/200 rate is a conditional diagnostic rather than a ceiling on verification ability. On four development-held-out schemas, Qwen3-VL-8B and Qwen3.5-27B generate SMT-equivalent criteria on all 256 primary calls per checkpoint yet still make 15 and 26 completed decision errors. Trace replay localizes 39/41 displayed error traces to incorrect arithmetic execution and the remaining two to incorrect transcribed inputs; none is a case where the terminal label contradicts the model’s displayed check. An authored executable checker and a restricted model-generated program baseline substantially improve pair accuracy. Together, these results turn a broad “read-versus-reason” failure into an auditable decomposition of verification and motivate evaluating verification as a pipeline rather than a single end-task score.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision-language models, multimodal verification, executable specifications, counterfactual evaluation, robustness, program-assisted reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ohWxOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper emphasizes that a critical point in evaluating vision-language models that the records were not being treated seriously. Therefore, it provided a benchmark for auditable verification ContraLedger, whose core design is “Counterfactual Twins”, “Executable Specs” and “Read-Know-Decide Protocol”. This paper proposed a fine-grained approach for failure analysis, as well as conducted empirical study and decomposition experiments.

Critical strengths
  • I tend to think the authors are targeting to an important aspect in visual scenes and they point to a very certain task.
  • The technical contributions well serve for the problem to solve (even though the results are not always positive). I also check the code package, which seems reliable.
Critical weaknesses
  • This paper is weak in presentation which takes long time to follow. I strongly suggest authors to revise the writing carefully, such as unify the terms and sharpen the storyline.
  • Based on the results, existing methods and ContraLedger seem lie at the same level, all show strong tendency to be misled by attacks. It seem an evidence showing ContraLedger is not good enough to prevent the attacks? Maybe you aim to prove verification failures is common and hard challenges? Then, ContraLedger cannot be regarded as a contribution, right?
  • The failure results seems sensitive to response biases, and inference protocols, etc. Therefore, the visual failure still remains insufficiently isolated. It makes me a bit doubt on the broader significance.
Questions for authors

I am not sure you are actually doing exactly scene text understanding, for example, simply reproduce the known arithmetic reasoning errors in a visual-text setting (instead of raising multimodal verification)? How to tell the differences in your experimental settings?

1 comment
AuthorsOpenReview ↗

Response on Presentation, Matched Controls, and Verification Errors

Thank you for the presentation and interpretation suggestions.

1. Presentation and contribution. We rewrote the Abstract, Introduction, contributions and Discussion, unified terminology and organized the main text into three questions about matched presentations, prerequisite controls and executable error audits. We expanded the explanations of experimental contrasts, denominators and result interpretation. Similar carrier results support a shared verification problem, without showing an attack advantage for our carrier (Table 2). ContraLedger is an evaluation protocol connecting validity twins, executable labels, rule-equivalence checks and error replay.

2. Matched image/text results. We evaluated the same 128 pairs in both media. All entries below are correct pairs /128; Qwen3 denotes Qwen3-VL-8B and Qwen3.5 denotes Qwen3.5-27B (Table 3).

MethodQwen3 imageQwen3 textQwen3.5 imageQwen3.5 text
Structured633411694
Criterion-first114125101105
Generated program83125119128
Read+checker126128127128

Within each method, fields, public rules/assumptions, formats, decoding and stage-wise caps match. Other methods allow 256 output tokens; criterion-first allows 128+256 in both media. All 1,024 blind criteria are equivalent, yet criterion-first makes 3/256 and 23/256 completed text errors with exact fields in those same responses.

The image/text effect varies by method, and both criterion-first contrasts are non-significant after Holm correction. Text also changes representation and input length, so these results do not isolate a uniquely visual cause. Replay of 41 archived errors finds 39 arithmetic errors and two incorrect-input cases; each verdict agrees with its displayed CHECK. Figure 6a illustrates one replayable arithmetic error.

3. Bias and inference protocol. Full-set pair accuracy requires both twins correct and penalizes constant acceptance or rejection. Conditional rates can still reflect acceptance bias. Read/Know are independent calls and do not establish field use in Decide. Figure 5 and Appendix S show descriptive within-decision attention, without establishing exact reading or causal use. We distinguish completed errors, unresolved outputs and actual costs in Tables 42-43, and retain these attribution limits explicitly.

Reviewer_PtyDOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper identifies a failure mode in which VLMs can extract information correctly from images but fail to apply the verification rules. To diagnose this failure mode, it introduces an auditing protocol ContraLedger by using stage-wise decomposition。

Critical strengths

The proposed protocol provides an interpretable and auditable framework for VLM verification error attribution.

Critical weaknesses

Although the paper identifies and analyzes the verification failure mode, it does not propose any solution to solve this issue, which limits its technical novelty.

The protocol and its executable checker rely on predefined rules for verification, which may not cover broader visual reasoning scenarios, especially the open-ended cases.

The hard-coded rules used limits the generalizability and practical applicability of the proposed verification framework, and increases the annotation cost for extra rules at the same time.

Questions for authors

How could the proposed verification protocol be used in broader real-world scenarios, especially when the verification tasks do not fit the predefined rules or schemas used in the paper? Any additional annotations for the rules are needed?

2 comments
AuthorsOpenReview ↗

Response on Executable Repair, Generalization, and Specification Costs

Thank you for raising the intervention and applicability questions.

1. Contribution and concrete interventions. ContraLedger connects matched validity twins, executable labels, criterion-equivalence checks and trace replay to localize observable verification errors. We also evaluate interventions on the same 128 image pairs. Entries below are correct pairs /128 from the fresh matched replication (Table 3).

MethodQwen3-VL-8BQwen3.5-27B
Structured63116
Criterion-first114101
Generated program83119
Read+checker126127

On the same completed records, Read+checker repairs 65 and 11 Structured errors with zero newly introduced errors (Table 42). Unresolved outputs count against full-set pair accuracy. The checker receives an extra authored rule; the generated-program arm receives field descriptions and assumptions without a hand-coded validity formula.

2. Generalization. We test four development-held-out schemas covering speed, DC power, kinetic energy and density (Appendix O). On the same frozen pairs, InternVL2-8B produces 244/256 equivalent criteria, followed by 122 completed errors, including 108 with exact same-response fields. However, its Read+checker reaches only 26/128 correct pairs, with 179/256 unresolved outputs (Appendix P.1). Thus diagnostic evidence extends across schemas/model families, while intervention transfer remains sensitive to extraction. Open-ended and naturally captured-record transfer are unvalidated.

3. Annotation and real-world scope. A new audited schema requires field definitions, units, domains, tolerances, assumptions and a validated reference rule, plus record references and image/semantic quality checks. Generation automates nominal labels after this specification; it does not eliminate annotation cost. Generated programs reduce manual formula provision, while evaluation still needs validated references.

The protocol fits tasks with explicit contracts, such as conversion records or conservation ledgers. Where validity cannot be operationalized, human adjudication or another task-specific evaluation is needed. We clarify these boundaries in the Discussion and retain independent human semantic validation as an open limitation.

Reviewer_PtyDOpenReview ↗

Response to author rebuttal

Thanks for the reviewer's feedback. It partially addressed my concerns, but I feel the answer is still a bit superficial in general and overall concerns to the paper still exist. I will maintain my score.

Reviewer_CjvCOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper addresses the problem that vision–language models may produce incorrect judgments even when they can accurately read numerical fields and express the relevant verification rule. The authors propose ContraLedger, which pairs valid records with invalid records obtained by changing a single field. They evaluate verification as a pipeline of extraction, criterion specification, completion, arithmetic execution, and final judgment. The experiments analyze errors across multiple models and additional schemas, and compare structured prompting, executable checkers, and model-generated programs.

Critical strengths
  • Distinguishing the ability to read a record from the ability to verify it is well motivated and practically relevant to reliable VLM evaluation.
  • Constructing valid and invalid records that differ in one field, and auditing their construction and the model's decisions against executable specifications, is appropriate for analyzing failure sources. Separating extraction, criterion expression, and execution from end-task accuracy is another positive aspect.
Critical weaknesses
  • The additional contribution relative to existing analyses of arithmetic reasoning errors needs clearer justification. Most Qwen errors in Appendix O.4 are explained by incorrect arithmetic rather than a terminal judgment detached from the displayed check. The authors explicitly state that a reasoned, budget-matched text-only baseline was not run. The residual errors on the additional schemas therefore do not sufficiently distinguish problems introduced by visual processing from general calculation errors.
  • The headline result of 191/200 is conditional on passing the source, valid-record, Read, and Know controls, rather than an error rate over all cases. Read and Know are also independent calls, so their success does not establish that the same information was used in the Decide call. Section 6.3 acknowledges this limitation, but the claims supported by these probes versus the later same-response analyses should be distinguished more clearly.
  • Prompting, completion, and computation are not controlled identically in every comparison. In Table 28, Qwen3.5-27B improves from 13/128 to 115/128 correct pairs, but unresolved outputs also decrease from 184/256 to zero. Meanwhile, criterion-first in Table 3 uses an additional call. Although equal-cap comparisons are included, further analysis is needed to establish how much each method reduces completed errors when actual token usage and latency are considered.
  • It is unclear whether the benchmark's scale and quality assurance sufficiently support its central contribution. The core evaluations contain 200 scenes each from COCO and VOC, while the additional schemas comprise only 128 valid/invalid pairs across four types. Several diagnostic studies reuse the same records, so model-call counts should not be interpreted as independent dataset size. Although the authors recompute rule labels and perform some SMT checks, they explicitly state that independent human validation of semantic validity, physical assumptions, and visual naturalness remains open. The post-run correction of 16 reference labels for derived-answer questions and the unavailability of some primary image assets and raw journals further motivate independent quality assessment and a clear account of release completeness.
Questions for authors
  • Can the authors provide text-only results on the additional schemas in Appendix O using identical fields, rules, output formats, and reasoning budgets? This would help isolate errors introduced by the image input.
  • Can the authors provide a single table showing the full sample size, the number passing each control, and the final errors for each model behind the 191/200 result? Please distinguish independent Read/Know probes from exact fields within the decision response.
  • After ensuring adequate completion, do the comparisons among neural prompting, generated programs, and authored checkers hold when actual token usage and latency are considered?
  • Can the authors provide deduplicated counts of source images, record pairs, and questions across all studies, together with their overlap across splits? Please also provide independent validation of the rendered images and questions for semantic validity and explicit assumptions, beyond the mathematical validators, and clarify plans to release the complete evaluation assets.
3 comments
AuthorsOpenReview ↗

Response to Questions 1 and 2

1/4. Matched image/text control and contribution

We completed the requested reasoned image/oracle-text comparison on the same 128 pairs. Q3 denotes Qwen3-VL-8B and Q35 denotes Qwen3.5-27B. Pair counts are correct valid/invalid pairs out of 128; CE is wrong decisions / completed records.

ModelMethodImage pairsText pairsImage CEText CE
Q3Structured633467/256100/256
Q3Criterion-first11412514/2563/256
Q3Generated program8312516/2050/252
Q3Read+checker1261282/2560/256
Q35Structured1169412/25634/256
Q35Criterion-first10110527/25623/256
Q35Generated program1191289/2560/256
Q35Read+checker1271281/2560/256

Within each method, fields, units, public rules/assumptions, formats, decoding and stage-wise caps match across media. Text supplies all carrier strings without an image. Structured, program and checker use a 256-token call; criterion-first uses 128 tokens for its blind rule call plus 256 for its decision. Unresolved outputs remain errors in full-set pair accuracy.

All 1,024 blind criteria are SMT-equivalent. Nevertheless, the 3 and 23 completed criterion-first text errors have exact fields in those same decision responses. Both criterion-first image/text contrasts are non-significant after Holm correction; representation effects vary across methods. Oracle text changes representation and input length, so this does not isolate a uniquely visual cause.

We also agree with your trace interpretation. Of 41 archived errors (Appendix O.4), 39 involve arithmetic and two involve incorrect inputs; all final verdicts agree with the displayed CHECK. We revised the contribution to the executable connection between one-field twins, validated labels, rule-equivalence tests and observable error replay. Figure 6a illustrates one replayable arithmetic error. These results appear in Table 3 and Appendix R.2.

2/4. Complete sample funnel and evidence from independent probes

We added the complete per-model funnel below (Table 40). For each model, N=200 shared scenes; the pooled row sums scene-model decisions. S and V are source-absence and valid-record successes. SV requires both; SVR additionally requires exact independent Read; SVRK additionally requires independent Know rejection. E is false acceptance within SVRK. FA and Pair use the full N, without prerequisite filtering.

ModelNSVSVSVRSVRKEFull FAFull Pair
Qwen2.5-VL-3B2001961791752719161793
Qwen2.5-VL-7B20019919419310570701933
Qwen3-VL-8B20019795933021179516
LLaVA-OV-8B2001601811414225231822
InternVL2-8B2002001991995530301970
InternVL3-8B2001661551365835351575
Pooled decisions120011181003937317200191100329

Thus, 191/200 is conditional on 200 selected scene-model decisions covering 113 distinct scenes, drawn from 1,200 decisions on 200 shared scenes. We removed this conditional headline from the Abstract and clarified the reporting unit in Section 5.2.

Read and Know are separate calls with no shared history with Decide; Know retains the clean source image. Passing these probes does not establish information use inside Decide. The later execution audits separately record fields within the decision response and replay its displayed calculation. They support observable extraction/execution checks, without identifying hidden computation. We now distinguish these evidence levels explicitly.

AuthorsOpenReview ↗

Response to Questions 3 and 4

3/4. Completed-error repair, actual tokens and latency

For Qwen3.5-27B, the historical 13/128-to-115/128 gain mainly reflects completion recovery: unresolved outputs fall from 184/256 to zero. Direct has 5 errors among 72 completed records, versus 13/256 for Structured.

Fresh image results follow. Q3/Q35 denote Qwen3-VL-8B/Qwen3.5-27B. CE is wrong/completed; U is unresolved /256. Tokens and seconds are per record; both criterion-first calls count.

ModelMethodCEUInput/output tokensMean/p95 seconds
Q3Structured67/2560634.1/93.02.88/3.67
Q3Criterion-first14/2560899.1/120.53.73/4.38
Q3Generated program16/20551685.8/48.01.57/1.99
Q3Read+checker2/2560522.8/39.01.28/1.53
Q35Structured12/2560643.1/93.06.05/7.54
Q35Criterion-first27/2560916.3/109.57.24/8.27
Q35Generated program9/2560694.8/90.15.86/6.58
Q35Read+checker1/2560527.8/39.02.72/3.21

To separate correction from completion, we also compare the same records completed by Structured and each comparator.

ModelComparatorCommon nStructured/comparator errorsRepaired/introduced
Q3Criterion-first25667/1457/4
Q3Generated program20557/1650/9
Q3Read+checker25667/265/0
Q35Criterion-first25612/273/18
Q35Generated program25612/95/2
Q35Read+checker25612/111/0

Checking repairs 65 and 11 completed errors without introducing new errors. Criterion-first is model-dependent and increases completed errors for Q35. The common-completion subsets are descriptive; all unresolved outputs remain errors in full-set pair accuracy.

The authored checker supplies an extra correct rule. Timings use eight concurrent pipelines, one A6000 for Q3 and two for Q35, and exclude loading. They describe measured tradeoffs rather than compute-matched superiority. Tables 42-43 report these analyses; Table 43 additionally includes text-condition costs.

4/4. Deduplicated data inventory, quality assurance and release

We inventoried 60 recoverable manifest/case indices. The principal banks are shown below (Table 44). Counts are deduplicated within banks; overlapping banks must not be summed as independent data.

BankSourcesPair groupsQ
COCO development12012092
COCO held-out400400203
COCO confirmation200200158
VOC confirmation200200131
Fresh development6464110
Fresh test256256336
Derived answers, fresh subset128—120
Rule-explicit confirmation128—125
Media/association128192—
Original-schema budget study128128—
Four-schema parent128128—

Q counts distinct stored question/query strings, including diagnostic queries, rather than independent semantic tasks. A dash means the field is unavailable, not zero. The 192 media/association groups include swaps, not 192 independent scenes. This recoverable inventory is not an exhaustive reconstruction of all historical records.

The COCO development/held-out/confirmation source splits are disjoint. Fresh development/test do not overlap those banks or VOC confirmation; derived answers reuse 128 fresh test scenes. Original-schema diagnostics reuse 64 COCO and 64 VOC confirmation sources. The four-schema bank overlaps older COCO development/held-out/confirmation by 3/11/2 IDs. Model, program and media extensions, including the 5,120 new calls, reuse its 128 pairs. Accordingly, we claim development-held-out schemas, not globally unseen images, and do not count repeated calls as new data.

All 16 corrected temporal references are disclosed with old/new labels and reasons; raw responses are unchanged. Independent human ratings of legibility, semantic validity, assumption sufficiency and naturalness remain unavailable. Mathematical validators do not replace that assessment.

We prepared an image-free replay package containing saved calls, fields/labels, scoring code and audits. Anonymous access verification, image redistribution clearance and complete historical release remain unresolved. These limits and the overlap/correction ledgers are stated in Appendices R.3-R.4.

I appreciate the authors' rebuttal. Although not all concerns have been fully resolved, the responses to the key issues have been addressed. Therefore, I have updated my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.