acceptodds
Under review as a conference paper at ICLR 2027

When VLMs Trust Context: Evaluating Scene Text Recognition under Misleading Context

Abstract

Vision-language models (VLMs) can read text in natural scenes, but their predictions may be influenced by the surrounding context. When the printed text conflicts with what the scene suggests, a model may return a more plausible word instead of the shown text. We introduce SceneFaith, a benchmark of 781 generated scene images for studying this behavior. Each output is classified as Literal, Canonical, or Other, separating faithful transcription from context-consistent rewriting and ordinary recognition errors. Across 15 models from seven families, all models show rewriting on clear images, with rates ranging from 8.45% to 58.51%. Controlled experiments further show that surrounding context matters: removing surrounding scene information reduces rewriting and improves literal accuracy, while changing the scene around the same text patch can also change model outputs. Moreover, weakening the target text with blur increases rewriting. These results show that reliable scene-text recognition requires VLMs to balance visual character evidence with contextual information, preserving clear text while using context mainly when the visual evidence is uncertain.

Primary area:
datasets and benchmarks
Keywords:
VLM Evaluation, Vision-Language Models, Visual Text Recognition

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 5 reviews, above 27% of the others.
Reviewer_YWqzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SceneFaith, a benchmark of 781 generated scene images across 17 categories in which an unusual printed string conflicts with a contextually plausible conventional spelling. Outputs are classified as Literal, Canonical, or Other, and 15 model variants from seven families are evaluated. The main diagnostic contribution is to compare predictions after removing peripheral content and after changing the scene around an identical target patch in a separate 150-pair set. Additional experiments examine blur, lexical preference, model variants, consensus filtering, and a post-training pilot. These experiments document canonical substitutions and peripheral-input sensitivity on a deliberately constructed stress test.

Critical strengths
  1. The fixed-target-patch interventions are a useful addition to transcription-faithfulness evaluation. Holding the target pixels, position, scale, and local background fixed addresses an important limitation of comparing full images with enlarged crops, and provides evidence that peripheral input affects transcription.

  2. The Literal/Canonical/Other breakdown distinguishes recovery of the printed string from changes between different error types. The broad model panel and paired uncertainty estimates make the reported failure patterns informative. The consensus-filtering example also illustrates a practically relevant limitation of agreement-based validation.

Critical weaknesses
  1. The information gained beyond existing rewriting studies is limited. As the paper acknowledges, TextHalu-Bench and FaithC4 already document semantically plausible substitutions of unusual text. Confirming such substitutions, together with blur sensitivity and associations with lexical preference, provides limited incremental insight on its own. Holding the target fixed is a meaningful extension, but the conditions governing the observed context sensitivity remain unclear. The increase in rewriting after gray masking in Table 17 is potentially more informative. This uses a different target set from the main benchmark, so it is not a contradiction; however, the conditions governing the change in direction remain uncharacterized. Comparisons with OCR-specialized recognizers on the same targets would also help establish the test's distinctive diagnostic value. Overall, I find the stress test useful, but its demonstrated information gain and evaluation utility insufficient for acceptance.

  2. The validity of the intended context manipulation needs clarification. Section 3.2 states that the canonical spelling must not appear elsewhere in an image, yet Figure 2 contains "leopard" in the paragraph outside the box containing "ieopard." I cannot determine whether this example belongs to the evaluated 781 images. If it does, copying another visible word is a competing explanation, and gray masking removes this answer source along with scene content. This is a concrete concern about the stated screening procedure, although it does not establish dataset-wide leakage. Moreover, the A/B comparison changes identifier role and visual appearance alongside scene meaning, as Appendix H acknowledges. The experiments therefore establish peripheral-input sensitivity more securely than a specific semantic-context effect. This limits the interpretation of the paper's main extension beyond prior rewriting studies.

Questions for authors
  1. Is the Figure 2 example included in the 781-image benchmark? How was the rule excluding canonical spellings elsewhere in the image implemented, what fraction of evaluated images contain such text, and do the principal full-versus-gray effects remain on a verified leakage-free subset?

  2. Can the authors use the existing paired controls to characterize the conditions under which gray masking increases versus decreases rewriting? What evidence distinguishes scene meaning from identifier role, peripheral text, and other visual changes? This would clarify what new understanding the benchmark adds beyond documenting peripheral-input sensitivity.

  3. Please clarify effective sample counts and scoring. Table 1's InternVL3-38B blur percentages (26.15/60.51/13.33) are consistent with counts of 204/472/104 out of 780, but not integer counts out of the stated 781; its clear RR also changes from 58.51 in Table 1 to 58.46 in Table 15. Were responses excluded, and which common subset was used for paired analyses? Also, what deterministic rule implements "deletes unrelated text," particularly when an answer includes both the printed and canonical strings?

7 comments
AuthorsOpenReview ↗

Response to Reviewer YWqz

We thank the reviewer for their careful reading and for raising these thoughtful questions. We have conducted additional experiments to answer each of them.

W1. Incremental knowledge, the Table 17 reversal, and same-target OCR

What is not new. TextHalu-Bench and FaithC4 already document the replacement of unusual strings by plausible spellings; FaithC4 in particular separates recovery of the unperturbed word from other errors. §2.2 now describes this accurately, and we no longer present the phenomenon as a discovery.

What the interventions add. The fixed-patch diagnostic holds the target patch, including its local background, at identical pixels, position and scale while only the periphery changes. Across 11 aliases, mean RR is 6.85% in identifier scenes and 11.15% in scenes supporting the conventional word: +4.30 points, with 9/11 positive estimates and 5/11 pointwise 95% intervals above zero. This is a paired estimate of peripheral-input sensitivity with a stated per-model spread, not a further demonstration that rewriting occurs.

The Table 17 reversal. We report the reversal with intervals instead of explaining it away. On the 150-patch set, gray-minus-aligned RR differences are +17.33 points [10.67, 24.67] for Qwen3-VL-8B Instruct, +20.00 [12.00, 28.00] for Qwen3-VL-235B-A22B Instruct, and −14.00 [−20.67, −7.33] for Qwen3-VL-32B Thinking. On the 781-image main set, masking lowers RR for all four models tested (6.53–20.10 points clear, 7.68–25.86 under blur). The sign therefore depends on the model, even on a shared set of 150 targets.

We do not have evidence identifying why the sign flips. The two datasets use different targets and a different rendering of the periphery, and the mask removes semantics, nearby text and clutter together. Consequences for the paper: we no longer treat the mask as a neutral "remove context" operation, we describe masking as a manipulation whose effect on rewriting is model- and stimulus-dependent, and the reversal appears in the results discussion rather than only in an appendix table. We consider this an open question the benchmark raises rather than resolves, and we have marked it as such.

Same-target OCR. HunyuanOCR-1.5[1] on the 150 unique patches: 98.67% L, 1.33% C. On clear crops: 91.68 / 3.46; on moderate-blur crops: 83.99 / 10.24. The three STR systems (PARSeq[2], ABINet[3], CRNN[4]) on clear crops span 24.46–84.25% L and 2.30–7.94% C. Given the target region, a dedicated recognizer recovers the printed string much more often, and it still substitutes the conventional spelling at a low rate. This gives the reference point requested: what a system that is not making scene-level inferences does on these strings.

Contribution, restated. The paper now claims three specific measurements rather than the phenomenon: the paired intervention with effect size and heterogeneity, the divergence between RR ranking and literal-accuracy ranking, and the comparison showing that dedicated recognizers substitute the conventional spelling at lower but nonzero rates.

W2. Validity of the context interventions

Figure 2. We confirm the example is the benchmark image full500_004_animal_A. The figure's PNG matches the evaluation file by SHA-256, and leopard appears twice in the paragraphs outside the target box. In the original pairing, all four models output leopard on the full image; with the gray mask, GPT-5.2 still outputs leopard and the other three output ieopard. The reviewer's alternative explanation — direct copying of a word visible elsewhere in the image — is correct for this example.

How often this happens. We re-screened all 781 images for the exact canonical spelling outside the target box. It occurs in 22 images (2.82%); a further 9 contain only simple inflections, and 6 are uncertain. Combined with the exact cases, the wide criterion gives 31 images (3.97%), and including all uncertain cases gives 37 (4.74%).

Does the main conclusion survive? On the 744 images that screen negative for exact and inflected canonical spellings, the Gray − Full RR differences are −18.15 (GPT-5.2), −5.65 (Gemini 3.5 Flash), −9.01 (Claude Sonnet 4.6) and −19.89 (Qwen3-VL-32B Thinking) points, with all four paired intervals below zero. On the full 781 these are −18.57, −6.53, −9.60 and −20.10. The shrink is modest and the direction is unchanged, so copying cannot account for the overall masking effect. The leak is real, but small.

Framing. The reviewer is also right that A/B changes scene semantics, target role and visual appearance together, so the experiments identify sensitivity to peripheral input and not a semantics-specific effect. That is now the paper's claim, and the limitation is stated where the result appears and again in the Limitations section.

AuthorsOpenReview ↗

Response to Reviewer YWqz

Q1. Is Figure 2 in the benchmark, how is the no-leak rule enforced, what fraction of images contain such text, and does the main effect survive excluding them?

This question has four parts; we answer each.

  1. Figure 2 is in the benchmark (full500_004_animal_A), with two leopard instances outside the box. It should not have passed screening, and we say so in the paper rather than replacing the figure silently.
  2. Prevalence. 22 of 781 images (2.82%) contain the exact canonical spelling outside the box; 9 contain only simple inflections; 6 are uncertain. The full-positive upper bound is 37/781 (4.74%).
  3. Enforcement. The original cohort's scene screening by Gemini 3.1 Flash Lite treated complete canonical words, case variants and simple inflections as exclusion criteria; the crop blind read checked transcription of the target only. The expansion added a GPT-5.5 full-image check for occurrences of either spelling outside the box, with AI-assisted review. The miss in the original cohort shows the check was not exhaustive — the model reported no outside spelling for this image. For this revision we ran a full re-screen (model tiling at native resolution plus Apple Vision OCR with language correction disabled, with visual confirmation of every candidate) rather than a spot check, and report the result.
  4. Subset recomputation. On the 744-screen-negative set, the Gray − Full RR difference remains negative for all four models with intervals excluding zero (values above). The sensitivity variant that also excludes the uncertain cases (753 pairs) leaves the differences essentially unchanged (−18.19, −5.98, −9.30, −20.05).

Q2. When does masking increase rewriting, and can semantics, identifier role, peripheral text and visual changes be separated?

When the mask increases rewriting. On the 150-patch set, for both Qwen3-VL Instruct aliases (8B and 235B-A22B) masking produces markedly higher RR than either full-scene condition — for 8B Instruct, 25.33% on gray against 8.00% aligned and 4.00% identifier. For Qwen3-VL-32B Thinking, masking lowers RR relative to the aligned scene (8.00% vs 22.00%). On the 781-image main set, masking lowers RR for all four models.

Separation of factors. What the current controls can separate: the target patch's contribution, which is held fixed in both the mask experiment and the A/B experiment, and the joint contribution of the periphery, which is measured by both. What they cannot separate: the individual contributions of scene meaning, word-versus-identifier role, nearby text and low-level appearance. Each intervention changes at least three of these at once. The same-domain shuffle narrows the gap for scene meaning but not for appearance. We do not claim to identify semantics, and the paper states that an appearance-matched, role-matched semantic control would be required to do so, which the current image set does not support. This is a real limitation of the diagnostic design.

Q3. Denominators, pairing subsets, and the parsing rule

Denominators. The provider API returned HTTP 400 for one moderately blurred input, nataexp_0138. InternVL3-38B's Table 1 moderate-blur counts are therefore 204 L / 472 C / 104 O, out of 780. No other responses were excluded from Table 1.

Table 1 vs Table 15. Table 15 pairs clear and blurred responses by image ID and uses the 780 IDs with valid responses in both conditions. For InternVL3-38B the clear RR is 457/781 = 58.51% in Table 1 and 456/780 = 58.46% in Table 15. The difference is the single missing ID, not a different sample.

Parsing rule (deterministic, now in §4.1). Remove an enclosing code fence; keep the first nonempty line; remove a leading Answer:, Output:, Text: or Result: label (case-insensitive) and one pair of matching outer quotation marks or backticks; fold case and normalize whitespace. The entire retained line must match the printed form (L) or the canonical form (C); otherwise O. Both spellings on the retained line → O. Both spellings on separate lines → only the first nonempty line is scored. We also report, in Appendix M, all 15 aliases rescored under strict whole-answer matching: 92/11,715 clear and 60/11,714 moderate answers change label; the moderate RR ranking is unchanged, the clear one has one adjacent swap and one resolved tie; all eight Full − Gray RR differences are identical under both rules.

References

[1] HunyuanOCR-1.5: Making Lightweight OCR VLMs Faster and Better

[2] Scene Text Recognition with Permuted Autoregressive Sequence Models

[3] Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition

[4] An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition

Thank you for the response and for the new 2B study. I still have two concerns.

  1. Section 4.6 says the 4B pilot trains on 300 clear and blurred pairs and is evaluated on 781 test pairs, which is the whole benchmark. If the 300 pairs come from SceneFaith, they must overlap the 781 evaluated images, since 300 plus 781 is larger than 781. Where do the 300 pairs come from? Are they separate from the 781 images?

  2. In your 2B table, continued SFT reaches a CER of 16.59 and GRPO reaches 20.35, so extra steps alone do more than GRPO, and GRPO is worse than SFT on CER, 20.35 against 19.24. However, Section 4.6 still reports a modest but consistent gain over SFT from a comparison against a shorter trained arm.

Reviewer_d1oKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SceneFaith, a benchmark of 781 generated scene images containing deliberately perturbed text. It evaluates 15 VLM variants using Literal/Canonical/Other outcomes. Through masking, fixed-target-patch comparisons, cropping, and blur experiments, the paper studies how surrounding context affects transcription. A small SFT and GRPO experiment explores possible mitigation.

Critical strengths
  • The paper studies a practical problem: models sometimes replace visible text with a word that better fits the context, even when literal transcription is requested.
  • Changing the surroundings while keeping the text pixels fixed is a useful experimental design. It provides convincing evidence that surrounding visual input can affect transcription.
  • The experiments cover several model families and include paired confidence intervals and family-weighting checks. The authors also acknowledge important limitations.
Critical weaknesses
  • The readability checks need more detail. It is unclear who performed the blind readings, how samples were accepted or rejected, and how often independent readers agreed. This matters especially for blurred text, where a canonical answer may also reflect visual ambiguity. An independent human audit would strengthen confidence in the benchmark.
  • Scene meaning is still mixed with other changes. Masking removes clutter and nearby text, while the A–B comparison also changes the text’s role as a word or identifier. The shuffled-scene control helps, with positive differences for all eight models, although only two confidence intervals exclude zero. More closely matched controls would help isolate the semantic contribution.
  • The comparison with earlier work could be clearer. [FaithC4 already distinguishes rewriting from other recognition errors (Table 5)], so Section 2.2 should describe this more accurately. The paper should clarify what the scene interventions add and include dedicated OCR/STR baselines to better contextualize the results.
  • The training experiment needs clearer data splits and settings. Reporting three seeds is useful, but the source of the training pairs and their separation from the test set remain unclear. A continued-SFT baseline with a similar training budget would help explain what GRPO adds. Post-training L/C/O results would also show whether canonical substitutions specifically decrease.
Questions for authors

My main concerns are the readability validation, the added value over prior work, and the completeness of the training experiment. Addressing these points would make the contribution more convincing.

  1. Who performed the blind readability checks, and can independent agreement results be provided for clear and blurred text?
  2. Can scene meaning be varied while keeping the text’s word/identifier role and visual clutter approximately matched?
  3. How do dedicated OCR/STR systems perform, and which findings go beyond existing rewriting evaluations?
  4. How are training and test examples separated, and does GRPO still outperform continued SFT under comparable training budgets?
4 comments
AuthorsOpenReview ↗

Response to Reviewer d1oK

We thank the reviewer for their careful reading and for raising these thoughtful questions. We have conducted additional experiments to answer each of them.

W1. Blind readability checks and how samples were retained

Original cohort. The pipeline had two stages. Gemini 3.1 Flash Lite first screened paired scenes. It then blind-transcribed the clear target crops in a separate request that withheld the scene, the printed reference and the canonical spelling. A pair passed only if both crops contained one fully visible text span, both transcriptions matched exactly including case, and both confidence scores were at least 4/5. 356 pairs passed scene screening; 337 passed the crop check, and the benchmark contains their A images. Gemini also returned the canonical spelling for 6 l↔i images in that blind read, and all 6 were excluded — the expected direction of the filter, but also a reminder that the retained set is conditioned on passing a model's reading.

Expansion. GPT-5.5 transcribed clear crops at native resolution and separately screened full images for a single intact target inside one red box, scene support for the canonical concept, and occurrences of either spelling outside the box. An AI-assisted visual review then retained 444 of 1,000 candidates and reversed 135 of GPT-5.5's provisional decisions.

Independent readability check. Apple Vision with language correction disabled, on crops inset from the red box: 734/781 clear and 658/781 moderately blurred exact matches, including 30/31 and 18/31 of the l↔i targets.

Human read. One author reviewed 100 images spanning both clarity conditions: 98 fully readable, 2 partly readable, 0 unreadable.

Reader agreement. We do not have it. There was one reader and the ratings are pooled across conditions, so neither inter-reader agreement nor per-condition human transcription accuracy can be reported. We state this in Appendix L.1 rather than derive a substitute, and the per-condition evidence in the paper comes from the independent OCR check.

W2. Scene semantics still entangled with other changes

The criticism is correct as stated, and we have changed the claim rather than argue with it.

What changes together: gray masking preserves target pixels but removes nearby text, visual clutter and semantic cues at once. The fixed-patch A/B contrast changes scene meaning, the target's role as a word versus an identifier, and background appearance. The same-domain shuffle keeps the target patch and places it in a scene supporting a different word, but shuffled scenes are not appearance-matched.

Results under those constraints: in the shuffle, all eight aliases have higher RR in the aligned scene, and only the Instruct and Thinking variants of Qwen3-VL-235B-A22B have pointwise 95% paired intervals wholly above zero. Gray masking lowers RR by 6.53–20.10 points on clear images and 7.68–25.86 under blur for the four models tested.

What the paper claims: sensitivity to peripheral input, with per-model effect sizes and uncertainty, including heterogeneity (9/11 positive point estimates in A − B, 5/11 intervals above zero). What it no longer claims: that scene semantics is the identified cause. The Limitations section now states that an appearance-matched, role-matched semantic control is not available in this image set and that we do not have a semantics-specific estimate. We regard this as an unresolved limitation, not as something the existing controls approximate closely enough to substitute for.

AuthorsOpenReview ↗

Response to Reviewer d1oK

W3. FaithC4 comparison and dedicated OCR baselines

§2.2 correction. FaithC4 Table 5 distinguishes exact matches to the original, unperturbed word from other recognition errors at perturbed positions in the English-scramble condition. The submission's description was imprecise; §2.2 now states this accurately.

What the intervention adds. The fixed-patch diagnostic holds the target patch and its local background at identical pixels, position and scale while the periphery changes: mean RR 6.85% (identifier) → 11.15% (aligned), +4.30 points, 5/11 intervals above zero, and +4.24 with 4/11 under strict scoring.

Dedicated OCR on the same clear crops. HunyuanOCR-1.5[1] 91.68% L / 3.46% C; PARSeq[2] 68.63 / 5.12; ABINet[3] 84.25 / 7.94; CRNN[4] 24.46 / 2.30 with 73.24% O. General VLMs on the same crops: 86.94–96.93% L, 2.18–7.55% C. Dedicated recognizers are more literal, and they are not immune to substituting the conventional spelling. The clear-crop Other rate for CRNN[4] also illustrates why RR alone is not a fidelity measure.

W4. Training setup, budget control, and post-training L/C/O

Data. 300 groups of programmatically rendered learner misspellings from TOEFL-Spell, each in four views: related, unrelated, competing, gray. Train: 270 groups (1,080 images). Dev: 30 groups (120 images). All four views of a group stay in one split, and word families linked by Porter stem of the printed or canonical string are assigned together. Train and dev share 34 scene backgrounds, the gray control and 13 source essays, so dev is not independent of background or essay source. Against the 781 clear test images, checks find no overlap in exact printed strings, exact canonical strings, their Porter stems, or full-image SHA-256.

Training. Language-module LoRA (rank 16, alpha 32, dropout 0), vision module frozen, three epochs at 1e-4, target always the printed spelling. From each seed's SFT adapter, continued SFT and GRPO each run 600 seconds of CUDA-synchronized training-loop time on RTX 5090, including generation and reward computation for GRPO; branches stop after a complete optimizer step; model loading, preflight, export and evaluation are outside the budget. GRPO samples four completions per group at temperature 1, with a 64-token cap and no KL term, and reward 0.8 × whole-answer exact match + 0.2 × normalized character similarity, both against the printed string.

L/C/O on the 781 clear images:

ModelLCOCERSTR36 accuracy
Base45.5833.8020.6127.7787.92
SFT58.1320.4921.3819.2487.38
SFT → continued SFT58.0520.5721.3816.5987.61
SFT → GRPO59.1120.3220.5720.3587.32

Base → SFT reduces canonical substitutions by 13.32 points while O rises 0.77 points; the gain is predominantly reduced rewriting, with a small shift into other errors. Against continued SFT at matched budget, GRPO changes L by +0.77, +2.56 and −0.13 points across seeds 0–2, and CER is higher in all three GRPO branches. We report this as mixed evidence, not as a GRPO benefit.

AuthorsOpenReview ↗

Response to Reviewer d1oK

Q1. Who performed the readability checks, and can reader agreement be reported separately by clarity?

The blind checks were performed by models: Gemini 3.1 Flash Lite for the original cohort's crops, GPT-5.5 for the expansion's crops and full images, followed by AI-assisted visual review for final retention. Both withheld the scene, the printed reference and the canonical spelling.

The additional evidence requested here has two forms. The OCR check (Apple Vision, language correction disabled) is independent of screening and of the evaluated aliases, and it is reported separately by condition: 734/781 clear, 658/781 moderately blurred, with 30/31 and 18/31 for l↔i. The human read covers 100 images: 98 fully readable, 2 partly readable, 0 unreadable.

We cannot report human reader agreement or per-condition human transcription accuracy. We say this where the number appears rather than leave the gap implicit.

Q2. A control that changes scene semantics while matching text role and visual clutter

Constructing scenes that change what the scene implies while holding the target's word-versus-identifier role, the amount of nearby text, and visual complexity approximately fixed requires a new controlled image set, which we did not build for this revision. The same-domain shuffle is the closest available comparison and it is not appearance-matched: it changes surrounding text and visual appearance along with meaning, and only two of eight intervals exclude zero.

What we have done instead is narrow the claim to peripheral-input sensitivity, report each contrast with its confounds stated (mask removes semantics + nearby text + clutter; A/B changes meaning + role + appearance; shuffle changes meaning + surrounding text), and add the missing control to the Limitations section as an explicit item for follow-up work, together with the protocol sketch in Appendix C.3 (helpful-context evaluation with matched clarity and reliability conditions).

AuthorsOpenReview ↗

Response to Reviewer d1oK

Q3. Dedicated OCR/STR performance, and what goes beyond prior rewriting evaluations

W3 reports the clear-crop comparison for HunyuanOCR-1.5[1], PARSeq[2], ABINet[3], and CRNN[4]. The full comparison also includes moderate-blur crops and fixed patches. The finding that goes beyond "rewriting happens" is two-part: (i) the fixed-patch intervention gives a paired estimate with heterogeneity and uncertainty rather than an existence claim, and (ii) rewriting rate and literal accuracy order models differently, so a single rate cannot measure fidelity. The inverted orderings (GPT-5.5 vs Qwen3-VL-8B Instruct at equal 19.46% RR; GPT-5.2 vs Qwen3-VL-32B Instruct in the opposite direction) are in §4.2 and do not depend on any intervention.

Q4. How training and test samples were separated, and whether GRPO still helps at a comparable training budget

The evidence is the Qwen3-VL-2B-Instruct experiment: training on programmatically rendered learner misspellings from TOEFL-Spell, evaluation on the 781 clear SceneFaith images and six standard STR benchmarks.

Split. 300 misspelling groups, four views each (related, unrelated, competing, gray). All views of a group stay in one split, and groups linked by the Porter stem of either spelling are assigned together: 270 training groups (1,080 images), 30 development groups (120 images). Train and dev share 34 backgrounds, the gray background and 13 source essays, so dev is diagnostic only.

Test-set checks. No overlap with the 781 clear images in exact printed strings, exact canonical strings, their Porter stems, or full-image SHA-256. These are string- and file-level checks; they do not show that semantic families or background pixels are disjoint, and the paper says so.

4B pilot. Its training manifest and per-sample predictions were not retained, so that split cannot be verified at the source-image level. It stays in Appendix C.1 with this limitation and carries no split claim.

Matched budget. For each of three seeds, continued SFT and GRPO start from the same SFT adapter on the same training set, with 600 seconds of CUDA-synchronized training-loop time each on one RTX 5090, GRPO generation and reward included. Branches stop after a complete optimizer step; loading, preflight, export and evaluation are outside the budget.

GRPO − continued SFT on the clear set: ΔL +1.07 points on average (+0.77, +2.56, −0.13), with only the seed-1 interval excluding zero and seed 2 reversing; ΔC −0.26; ΔCER higher in all three seeds (+3.01, +3.38, +4.87), only seed 2 excluding zero. The reward is also near-saturated: 1,246 of 1,249 sampled groups have zero within-group variance, so GRPO has little to act on. We report a seed-dependent literal-rate gain, not a GRPO benefit.

Cost outside the target task: STR36 accuracy on the six standard benchmarks goes 87.92% (Base) → 87.38% (SFT) → 87.32% (GRPO), all SFT-minus-Base intervals below zero. Ordinary scene-text recognition is not preserved losslessly.

References

[1] HunyuanOCR-1.5: Making Lightweight OCR VLMs Faster and Better

[2] Scene Text Recognition with Permuted Autoregressive Sequence Models

[3] Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition

[4] An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition

Reviewer_9J4aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SceneFaith, a benchmark of 781 generated images that tests whether VLMs faithfully copy printed text when the surrounding scene suggests a different spelling. Experiments show that surrounding input affects transcription, while a small training study explores whether SFT and GRPO can improve accuracy.

Critical strengths
  1. The paper defines a useful stress test for faithful transcription. The task directly tests whether models preserve unusual printed strings when a more familiar spelling is suggested by the scene.

  2. Reporting literal, canonical, and other outcomes together makes the results easier to interpret.

Critical weaknesses
  1. The benchmark uses generated text and model-assisted screening, but it does not report an independent readability audit with agreement between readers. This matters especially for similar characters, where a canonical answer may reflect an unclear glyph rather than a failure to preserve clearly visible text.

  2. The evaluation does not include a dedicated OCR or scene text recognition baseline on the same targets. Such a comparison would help establish how much of the difficulty is specific to general-purpose VLMs.

  3. The main evaluation uses one neutral prompt and does not test an explicit instruction to preserve unusual spellings. It remains unclear how much rewriting can be reduced through a simple change in the task instruction.

  4. All benchmark images come from one image generator and are selected to create a conflict between text and context. The model rankings and effect sizes have not been checked on real captured scenes, which limits their practical scope.

  5. The blurred targets lack a complete independent readability check. This makes it difficult to separate unnecessary correction from cases where the visual information needed to distinguish the two spellings has been lost.

  6. The rules for removing unrelated text before scoring are not specified, and the strict whole-answer check is limited to the Qwen comparison. It remains unclear how output parsing affects the full model rankings and the measured effects of context.

Questions for authors
  1. What exact rules remove unrelated text before assigning L, C, or O, especially when a final answer contains both the printed and canonical forms?

  2. How does the claimed contribution differ from the error analysis already reported by FaithC4?

  3. How do dedicated OCR or scene text recognition models perform on the same target crops?

  4. How much rewriting remains when the prompt explicitly asks the model to preserve unusual spellings and avoid spelling correction?

  5. Does the GRPO gain remain against an SFT control with the same training budget?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 9J4a

We thank the reviewer for their careful reading and for raising these thoughtful questions. We have conducted additional experiments to answer each of them.

W1. Independent readability and agreement between readers

We have two independent checks, and we say plainly which part of the request remains unmet.

External OCR. Apple Vision with language correction disabled, reading crops inset from the red box and receiving nothing but pixels. It reproduced the printed string on 734/781 clear and 658/781 moderately blurred targets, including 30/31 and 18/31 of the single-character l↔i substitutions. On the archived crops without the extra inset the counts are 727/781 and 639/781, with 30/31 and 17/31 for the substitutions.

Human read. One author reviewed 100 images spanning clear and moderately blurred conditions: 98 fully readable, 2 partly readable, 0 unreadable.

This matters for the reviewer's reason for asking: if some apparent l↔i rewriting is visual ambiguity, the OCR-passing subset bounds how much. That subset shows rewriting persisting (W5), and we also report individual cases where ambiguity is the plausible explanation, such as softbali, where the final dot cannot be resolved after blur and all four crop models return softball.

W2. Dedicated OCR and STR on the same targets

This gap is closed. HunyuanOCR-1.5[1] and three STR systems were scored on identical frozen crops with the same whole-answer rule. Percentages are L / C:

ModelClear crop (781)Moderate-blur crop (781)Fixed patch (150)
HunyuanOCR-1.5[1]91.68 / 3.4683.99 / 10.2498.67 / 1.33
PARSeq[2]68.63 / 5.1278.75 / 7.3082.00 / 5.33
ABINet[3]84.25 / 7.9477.98 / 10.1280.00 / 13.33
CRNN[4]24.46 / 2.3027.40 / 3.2037.33 / 8.67

Four general VLMs on the same clear crops: 86.94–96.93% L, 2.18–7.55% C.

Interpretation, stated in the paper: HunyuanOCR[1] is more literal than any general VLM and falls inside the VLM range on C; the three STR systems are less literal. CRNN[4]'s low C is not fidelity — 73.24% of its outputs are Other. The difficulty is therefore not unique to general VLMs, but it is markedly larger there: the clear-crop C ceiling for dedicated recognizers is 7.94%, against 7.55% for the strongest VLM and 0.64–2.18% for the strongest closed systems.

W3. Single neutral prompt

Covered by the prompt experiment in Q4 below. Summary, for GPT-5.2 and Qwen3-VL-32B Thinking: exact-copy instructions lower RR by 15.24 and 24.71 points on the same 781 images, with both paired intervals excluding zero, but residual RR of 14.34–19.72% remains. Prompt changes attenuate the effect without removing it. The exact-copy prompt constrains more than spelling, so we do not attribute the full effect to the spelling instruction.

AuthorsOpenReview ↗

Response to Reviewer 9J4a

W4. Generalization to real scenes

We accept this limitation without argument. All 781 images come from gpt-image-2 and were selected for a conflict between printed string and scene, so the rankings and effect sizes characterize this stress test. Section 6 and the Limitations section now state this where the rankings are discussed, and the abstract says "781 generated scene images" rather than leaving scope to the reader.

W5. Blurred targets and canonical rewriting

We report the comparison on targets an independent OCR could still read under blur.

  • Both-condition OCR subset: 649 targets passed the inset-crop check in both conditions and had valid responses from all 15 aliases. Mean full-image RR rose from 22.70% to 32.53% under moderate blur.
  • Archived-crop variant: 624 targets, RR 22.04% → 31.60% across 15 aliases; 623 targets, 3.21% → 6.62% across four crop aliases.
  • Shape-confusable l↔i subset: on the 17 targets that passed in both conditions, the four crop models returned the canonical spelling in 5/68 clear responses and 17/68 blurred responses.

What this establishes and what it does not: canonical rewriting increases even among targets a separate OCR system could read, so the blur effect is not purely a visibility artifact. It does not license a per-response claim; individual canonical answers on blurred targets can still come from ambiguity, and we say so in §4.4.

W6. Parse sensitivity of rankings and context effects across all models

We rescored every archived final answer under strict whole-answer matching.

  • Label changes: 92 of 11,715 clear answers and 60 of 11,714 moderately blurred answers move from L or C to O.
  • RR rankings: on clear images Qwen3-VL-235B-A22B Instruct and Thinking swap fourth and fifth place, and the GPT-5.5 / Qwen3-VL-8B Instruct tie resolves. On moderately blurred images the RR ranking is unchanged.
  • Literal rankings: local shifts in both conditions. For example, Qwen3-VL-8B Instruct moves from seventh to tenth under moderate blur, and Qwen3-VL-32B Instruct from twelfth to tenth on clear images.
  • Paired contrasts: all eight Full − Gray RR differences (four models × two clarity levels) are identical under both rules; the largest change in the corresponding literal differences is 0.13 points. Fixed-patch A − B moves from +4.30 to +4.24 points, with 9/11 estimates positive and 5/11 → 4/11 intervals above zero. Aligned vs shuffled moves from +3.33 to +3.25, with 2/8 intervals above zero under both rules.
AuthorsOpenReview ↗

Response to Reviewer 9J4a

Q1. Rule for removing irrelevant text before L/C/O assignment

We remove an enclosing code fence, if present, and keep the first nonempty line of the final answer. From that line we remove a leading Answer:, Output:, Text: or Result: label (case-insensitive) and one pair of matching outer quotation marks or backticks. After case folding and whitespace normalization, the entire retained line must match the printed form (L) or the canonical form (C); otherwise the answer is O. There is no substring search.

Both spellings on the retained line: the answer is O. With wresling printed and wrestling as the canonical form, wrestling (misspelled as "wresling") is O. Both spellings on separate lines: only the first nonempty line decides, so wresling followed by wrestling is L, and the reverse order is C.

Q2. Difference from FaithC4

FaithC4 evaluates perturbed text in rendered documents and, in Table 5, separates exact recovery of the unperturbed word from other errors at perturbed positions. We now describe it accurately in §2.2 and no longer present the phenomenon itself as new.

SceneFaith adds a paired scene-text intervention: the target patch, including its local background, holds identical pixels, position and scale while the surrounding scene changes. Mean RR differs by 4.30 points between identifier and semantically aligned scenes across 11 aliases (5/11 intervals above zero), and the gray-mask comparison shows the same kind of effect with the periphery removed rather than replaced. The comparison establishes sensitivity to peripheral input under fixed target pixels. But it does not isolate scene meaning from the target's role or from background appearance.

Q3. Dedicated OCR/STR on the same target crops

See the table in W2. Clear-crop L: HunyuanOCR[1] 91.68, PARSeq[2] 68.63, ABINet[3] 84.25, CRNN[4] 24.46; moderate blur: 83.99, 78.75, 77.98, 27.40; fixed patches: 98.67, 82.00, 80.00, 37.33. Same scoring rule and same crop files as the VLM crop runs.

Q4. Residual rewriting under an explicit preservation instruction

RR by condition on the same 781 clear images, for GPT-5.2 and Qwen3-VL-32B Thinking:

ModelNeutralExact-copyScene-ignoringCorrection-first (control)
GPT-5.229.5814.3415.3696.93
Qwen3-VL-32B Thinking44.4319.7224.5894.88

Neutral → exact-copy reduces RR by 15.24 points for GPT-5.2 and 24.71 points for Qwen3-VL-32B Thinking, both paired intervals above zero. Residual RR after the explicit instruction is still 14.34% and 19.72%: explicit preservation instructions reduce the effect roughly by half but do not remove it. Adding "ignore the scene" helps less than exact-copy wording (+1.02 and +4.86 points). The correction-first control shows both models can be steered to produce canonical answers on 94.88–96.93% of images.

Q5. GRPO versus continued SFT at matched training budget

New experiment: Qwen3-VL-2B-Instruct, three seeds. For each seed, continued SFT and GRPO start from the same SFT adapter, use the same training data and fresh optimizers.

On the same 781 clear SceneFaith images, GRPO minus continued SFT:

SeedΔLΔCΔCER
0+0.77 [−0.90, +2.43]+0.38 [−0.90, +1.66]+3.01 [−0.82, +9.99]
1+2.56 [+0.89, +4.35]−1.66 [−3.07, −0.26]+3.38 [−0.87, +11.17]
2−0.13 [−1.79, +1.54]+0.51 [−0.90, +1.92]+4.87 [+0.64, +12.30]

Mean ΔL = +1.07 points, ΔC = −0.26, ΔCER = +3.76. Only seed 1 has an L interval above zero; the seed-2 point estimate is negative; CER is higher in all three GRPO branches.

Our conclusion, now stated in the paper: at matched budget, GRPO yields a small, seed-dependent literal-rate gain, does not reduce canonical outputs on average, and costs CER. We do not claim a stable advantage over continued SFT.

References

[1] HunyuanOCR-1.5: Making Lightweight OCR VLMs Faster and Better

[2] Scene Text Recognition with Permuted Autoregressive Sequence Models

[3] Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition

[4] An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition

Reviewer_gEPAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a new benchmark called SceneFaith, which is made up of 781 scenes and are divided into 17 different categories and each one has a word that is spelled incorrectly on purpose - like "dolphiin" - inside a red box. The scene is designed to help the correct spelling make sense. When the results come out, they are put into three groups: Literal, which means the answer is exactly the same, Canonical, which means the answer is rewritten to use the correct spelling, and Other. The way they decide which group each result goes into is by using an exact match, and they don't use any human judges to help with the decision. They looked at 15 models and saw that the rewriting rate for clear images varied a lot - from 8.45% to 58.51%. On average, closed-source models rewrote 14.9% of the time, while open-source models did it 34.0% of the time. If masked the background or cropped the target, it reduced rewriting and the blur made rewriting more likely in every model.

Critical strengths
  • I like the way metrics has broken down into Literal, Canonical, and Other categories - it's straightforward and helpful. For instance, it's now clear when the model is autocorrecting a word versus simply misreading it. The experiments are also well-designed, especially how it kept the target pixels the same while changing the background. This approach makes it easy to see what's going on.

  • This finding is important in real-life situations. Even the most advanced models can alter text that is clearly printed when they are specifically instructed to copy it, and the more complex models can be even more prone to errors.

Critical weaknesses
  • My biggest concern is the data. All 781 images come from a single image generator, and nothing is tested on real photos. Images were screened with help from GPT-5.5 and a GPT/Claude/Gemini panel, the same families that score best, so I worry the benchmark is tilted in their favor. There’s also no human check that the text is actually readable. This matters for look-alike edits such as l↔i, where a “rewrite” might simply be genuine ambiguity, and for the blurred images, which were never checked at all.

  • The context effect is smaller than the paper suggests. In the cleanest experiment, the fixed-patch pairs, rewriting goes up by only about 4 points, and only 5 of 11 models show a clear effect. Gray masking removes both meaning and visual clutter, so it can’t tell us which one matters.

  • I was surprised that no traditional OCR systems or OCR-specialized VLMs were tested. The introduction argues that they are more faithful, so this is the comparison I most wanted to see. Simple prompt variations and existing mitigation methods are also missing.

Questions for authors
  1. How do conventional STR systems and OCR-specialized VLMs perform on SceneFaith? This directly tests the paper’s motivating claim.

  2. Does a stronger instruction (e.g., “the text may be misspelled; copy it exactly”) substantially reduce rewriting? If so, how much of the effect is prompt-dependent?

  3. Was a model from the evaluated families (GPT-5.5, GPT-5.2, Claude, Gemini) used to accept or reject images? If so, how do results change on images that pass a human or model-independent readability check?

5 comments
AuthorsOpenReview ↗

Response to Reviewer gEPA

We thank the reviewer for their careful reading and for raising these three thoughtful questions. We have conducted additional experiments to answer each of them.

W1. Data source, screening and readability

Single generator. All 781 images come from gpt-image-2. There is no real-photo or cross-generator evaluation, and we now state that the rankings and effect sizes describe a constructed stress test rather than general OCR behavior.

Screening by evaluated families. First, the screening process tended to retain images with more legible target text. Gemini 3.1 Flash Lite and GPT-5.5 participated in image screening. On 993 candidate crops with outputs from both recognizers, retained crops were easier to transcribe for both GPT-5.5 and the independent Apple Vision OCR system. The size of any model-specific advantage remains unquantified. We also observed the context effect in two models that did not participate in screening. Masking the surrounding scene reduced rewriting by 9.60 percentage points for Claude Sonnet 4.6 and 20.10 percentage points for Qwen3-VL-32B-Thinking.

Readability, including the shape-confusable characters. We ran an OCR check independent of the retention decisions and of the evaluated models: Apple Vision, language correction disabled, on crops inset from the red box. It reproduced the printed string on 734/781 clear and 658/781 moderately blurred targets, with 30/31 and 18/31 for the single-character l↔i substitutions. Separately, one author read 100 images spanning both conditions and judged 98 fully readable, 2 partly readable, 0 unreadable. The reviewer's specific worry — that some l↔i "rewriting" is genuine visual ambiguity — is partly correct, and we now say so.

W2. Effect size and the attribution of the scene interventions

We agree that the effect is modest in absolute size and that no single intervention isolates scene semantics. We disagree that this leaves the paper's claim unsupported: the claim is narrower than the one the review evaluates, and the interventions carry different confounds yet agree in direction.

Scope. We claim that peripheral visual input changes transcription when the target's own pixels, position and local background are held fixed, and that the change goes toward the conventional spelling. We do not claim a quantified causal effect of scene semantics alone. Section 4.3 no longer implies the mask isolates semantics.

Size. On the 150 fixed pairs, mean RR rises from 6.85% (identifier scenes) to 11.15% (scenes supporting the conventional word), +4.30 points, a 63% relative increase. It is not a legibility artifact: on the identical patches HunyuanOCR-1.5[1] reads 98.67% of targets literally. 9 of 11 aliases are positive; 5 pointwise 95% paired intervals exclude zero before multiplicity correction; GPT-5.2 is largest at +10.00 [4.00, 16.00]; two aliases are exactly zero. Under strict whole-answer scoring the mean is +4.24, still 9/11 positive with 4/11 intervals excluding zero. Nine of eleven positive signs is not what a null produces (sign test p≈0.03; aliases share images, so descriptive only). We report the per-alias distribution, not just the mean.

Convergence. The fixed patch holds target pixels, patch background, position and scale constant; the gray mask keeps target pixels and canvas geometry and deletes semantics, nearby text and clutter; the shuffle keeps a real scene and changes alignment only. The confounds differ, the direction does not: 9/11 positive (fixed patch), 8/8 positive (shuffle, 2/8 excluding zero), all four masked-model comparisons negative with intervals below zero. A semantics-only estimate is not available in these images; convergence across differing confounds is the strongest evidence the stimulus set supports.

Mask robustness. Gray masking lowers RR by 6.53–20.10 points (clear) and 7.68–25.86 (moderate blur), all paired intervals below zero. Excluding the 37 images with the canonical word or an inflection visible outside the box, the effect on the remaining 744 targets is −19.89 to −5.65, all intervals below zero. Two models that made no screening decisions show the same direction (Claude Sonnet 4.6 −9.60; Qwen3-VL-32B-Thinking −20.10), and the mask preserves target pixels, so the direction cannot come from altered target rendering.

An appearance-matched, role-matched semantic control is not available here; we state this as a limitation and make no semantics-specific causal claim. The remaining gap concerns attribution, not the existence or direction of the effect.

AuthorsOpenReview ↗

Response to Reviewer gEPA

W3. Dedicated OCR, prompt variants, and existing mitigation methods

Dedicated OCR and STR on identical frozen crops, same scoring. Percentages are L / C:

ModelClear crop (781)Moderate-blur crop (781)Fixed patch (150)
HunyuanOCR-1.5[1]91.68 / 3.4683.99 / 10.2498.67 / 1.33
PARSeq[2]68.63 / 5.1278.75 / 7.3082.00 / 5.33
ABINet[3]84.25 / 7.9477.98 / 10.1280.00 / 13.33
CRNN[4]24.46 / 2.3027.40 / 3.2037.33 / 8.67

On the same clear crops, four general VLMs score 86.94–96.93% L and 2.18–7.55% C. HunyuanOCR[1] falls inside both ranges; the three STR systems have lower L than every general VLM. CRNN[4]'s low C (2.30%) coexists with 73.24% O, so it is not more faithful overall. The dedicated systems are more literal, and they also substitute the conventional spelling, at lower rates.

Prompt variants. Three prompts plus a positive control, same 781 clear images, for GPT-5.2 and Qwen3-VL-32B Thinking. RR (%) by condition:

ModelNeutralExact-copyScene-ignoringCorrection-first (control)
GPT-5.229.5814.3415.3696.93
Qwen3-VL-32B Thinking44.4319.7224.5894.88

Neutral → exact-copy lowers RR by 15.24 and 24.71 points, both paired 95% intervals exclude zero, yet residual RR remains 14.34% and 19.72%. Prompting thus explains roughly half of the effect and cannot remove it. Adding an instruction to ignore the scene helps less than exact-copy wording in both aliases (+1.02 and +4.86 points). The correction-first prompt demonstrates that both models follow correction instructions reliably (94.88–96.93%), so neutral-prompt C outputs are a default preference, not a failure of instruction following. The exact-copy prompt bundles character-level copying with output formatting.

Existing mitigation. We evaluated ZoomText + Grounded Layer Correction on Qwen2.5-VL-3B on the same 781 targets under full-image input. Literal accuracy rises by 2.18 points on clear images [0.64, 3.71] and 1.92 points under moderate blur [0.38, 3.46]. C changes by +0.38 points on clear images [−0.77, 1.66] and rises by 2.05 points under blur [0.77, 3.46]. So the method recovers some literal accuracy without reducing rewriting.

AuthorsOpenReview ↗

Response to Reviewer gEPA

Q1. Dedicated STR and OCR-specialized VLM results

See the table in W3. Additional full-image figures for HunyuanOCR[1]: 86.30% L and 6.15% C on clear, 74.14 / 11.91 under moderate blur. The dedicated-vs-general comparison changes one of the paper's rhetorical claims, which we have adjusted: the failure is not exclusively a property of general VLMs. It is markedly more severe in them, with clear-crop C up to 7.55% against 2.30–5.12% for the three STR systems and 3.46% for HunyuanOCR[1]. All crop results describe the case where the target location is known.

Q2. Explicit preservation instructions and residual rewriting

See the prompt table in W3. Direct answers to the two parts of the question, for GPT-5.2 and Qwen3-VL-32B Thinking:

  • Does explicit instruction reduce rewriting substantially? Yes. Exact-copy instructions lower RR by 15.24 points for GPT-5.2 (29.58% → 14.34%) and 24.71 points for Qwen3-VL-32B Thinking (44.43% → 19.72%) on the same images, with both paired intervals excluding zero.
  • How much does the observed effect depend on the prompt? Substantially, but the dependence is partial. Instruction accounts for roughly half the rewriting, and residual rewriting of 14.34% and 19.72% survives explicit instructions to copy character by character. Changing the prompt attenuates the effect; it does not eliminate it.

Q3. Which models shaped the retained set, and what changes on independently verified images

Roles. Gemini 3.1 Flash Lite and GPT-5.5 participated in image screening, each in one cohort. For the original cohort, Gemini 3.1 Flash Lite screened the paired scenes and blind-transcribed the target crops; 337 of 500 pairs passed both checks, and the benchmark contains their A images. For the expansion, GPT-5.5 transcribed the target crops and screened the full images; an AI-assisted visual review then retained 444 of 1,000 candidates and reversed 135 of GPT-5.5's provisional decisions. GPT-5.2 and Claude Sonnet 4.6 did not participate in screening; both joined the consensus analysis only after the benchmark was frozen. In the blind crop read, Gemini 3.1 Flash Lite returned the conventional spelling for 6 l↔i images, and all 6 were excluded, so the original cohort is conditioned on passing the screening model's own reading.

Recomputation on verified images. Restricting to targets that Apple Vision transcribed exactly, and keeping only IDs with valid responses from all 15 aliases, mean RR falls from 26.35% to 25.20% on clear (734 targets) and from 36.95% to 32.65% under moderate blur (657 targets). The four lowest-RR aliases keep their order within each condition; some lower ranks shift. On the 649 targets that pass in both conditions, RR rises from 22.70% to 32.53% under blur.

Human check. One author read 100 images covering both conditions: 98 fully readable, 2 partly readable, 0 unreadable.

The honest summary: the ranking is robust to an independent readability criterion, the reading rate declines modestly on verified subsets.

References

[1] HunyuanOCR-1.5: Making Lightweight OCR VLMs Faster and Better

[2] Scene Text Recognition with Permuted Autoregressive Sequence Models

[3] Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition

[4] An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition

Reviewer_zuhkOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies context-induced rewriting in scene text recognition, where VLMs override visible character evidence and replace misspelled strings with contextually plausible canonical words. To systematically quantify this behavior, the authors introduce SceneFaith, a controlled benchmark of 781 generated images with deliberate visual-semantic conflicts, and propose a Literal/Canonical/Other evaluation framework that separates semantic rewriting from ordinary OCR errors. Evaluating 15 open- and closed-source VLM variants across seven families validates that all tested models exhibit substantial rewriting even on clear images (ranging from 8.45% to 58.51%), with reasoning-oriented ("Thinking") variants often amplifying the issue. Finally, a post-training pilot on Qwen3-VL-4B-Instruct using SFT and GRPO with an edit-distance-based reward demonstrates a viable mitigation pathway to improve transcription fidelity.

Critical strengths
  • The authors have well-controlled experimental design that isolates scene context from lexical priors. Rather than relying solely on correlation across full images, the paper uses gray-masking, cropping, and a 150-pair Fixed-Target-Patch set. This cleanly proves that peripheral visual context actively drives rewriting, while the L/C/O metric separates semantic over-correction from ordinary OCR failures.

  • The authors conducted comprehensive empirical evaluation across 15 VLMs. The benchmark uncovers non-trivial behavioral patterns, most notably that thinking variants in Qwen3-VL rewrite significantly more than their Instruct counterparts, and that rankings by rewriting rate do not align with literal accuracy.

Critical weaknesses
  • While the controlled interventions are valuable, the central phenomenon of semantic-prior-driven text rewriting has been explored in prior work. The paper does not yet sufficiently establish what fundamentally new insights its benchmark and analyses provide beyond existing studies.
  • The post-training evaluation is insufficiently connected to the paper's central failure mode. Section 4.6 reports improvements in EM and CER after SFT and SFT+GRPO, but omits the paper's core L/C/O breakdown, particularly Canonical Rewriting Rate. Consequently, it remains unclear whether the improvements reflect a reduction in semantic rewriting or merely fewer generic recognition errors. Furthermore, the evaluation does not examine whether training on perturbed text introduces trade-offs on standard scene-text recognition tasks.
Questions for authors

see weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer zuhk

We thank the reviewer for their careful reading and for raising these thoughtful questions. We have conducted additional experiments to answer each of them.

W1. What the benchmark adds beyond existing rewriting studies

We agree with the premise: FaithC4 already separates recovery of the original word from other errors at perturbed positions (its Table 5), and TextHalu-Bench already shows plausible substitution in scene text. Section 2.2 of the submission described FaithC4 imprecisely, and we have rewritten it to state what Table 5 actually reports, including the English-scramble condition. We no longer present "VLMs replace misspellings with plausible words" as a new observation. SceneFaith uses paired scenes with identical target pixels to test whether misleading context makes VLMs more likely to output the conventional spelling.

Three things in the paper are new relative to that work.

A paired intervention with pixel-identical targets. The 150-pair diagnostic holds the target patch, its local background, position and scale fixed, and changes only the surrounding scene. Across 11 aliases, mean RR is 6.85% in identifier scenes and 11.15% in scenes supporting the conventional word: +4.30 points, 9/11 positive point estimates, 5/11 pointwise 95% paired intervals above zero before multiplicity correction. Under strict whole-answer scoring this becomes +4.24 with 4/11 intervals. The gray-mask experiment gives an independent, opposite-direction result on the 781-image set: with target pixels unchanged, removing the periphery lowers RR by 6.53–20.10 points across four aliases. These experiments show that changing the surrounding visual context affects how often VLMs rewrite the printed text to its conventional spelling, even when the target pixels are identical. Neither comparison is available in FaithC4 or TextHalu-Bench, because both evaluate in the presence of a periphery they do not manipulate.

Rewriting rate and literal accuracy order models differently. GPT-5.5 and Qwen3-VL-8B Instruct both rewrite 19.46% of clear targets, but their literal accuracies are 78.10% and 62.23% (O rates 2.43% and 18.31%). The inversion also runs the other way: GPT-5.2 rewrites more than Qwen3-VL-32B Instruct (29.58% vs 27.53%) while reading more targets correctly (67.35% vs 60.95%). A benchmark reporting only EM or CER cannot distinguish these cases, and an intervention that lowers RR can still lower literal accuracy. This is a measurement point, not a rediscovery of the phenomenon.

Substitution is not exclusive to general VLMs. On identical frozen crops with identical scoring, the dedicated recognizers we added also produce the conventional spelling: HunyuanOCR-1.5[1] at 3.46%, PARSeq[2] at 5.12%, ABINet[3] at 7.94%, CRNN[4] at 2.30% (CRNN[4] with 73.24% Other). General VLMs on the same clear crops produce 2.18–7.55%. So the behavior is milder but present in systems trained to copy, and the gap between recognizers and VLMs is quantitative rather than categorical.

We accept that this is an incremental rather than a conceptual advance, and the contribution paragraph in the introduction and the section of §1 discussing novelty now say so.

AuthorsOpenReview ↗

Response to Reviewer zuhk

W2. Post-training: what the gains come from, and the cost to standard STR

The reviewer is right that the submission's §4.6 reported EM and CER without the L/C/O breakdown, and right that this leaves the source of the improvement unidentified. We have replaced the main post-training evidence with a new three-seed study on Qwen3-VL-2B-Instruct that reports exactly the missing quantities.

On the 781 clear SceneFaith images, percentages over all planned responses:

ModelLCOCERSTR36 accuracy
Base45.5833.8020.6127.7787.92
SFT58.1320.4921.3819.2487.38
SFT → continued SFT58.0520.5721.3816.5987.61
SFT → GRPO59.1120.3220.5720.3587.32

Trained rows average seeds 0–2; Base was evaluated once. STR36 accuracy is whole-word accuracy on six standard STR datasets totalling 7,672 images per version (CUTE80, IC13, IC15, IIIT5k, SVT, SVTP).

  • Where the SFT improvement comes from. Base → SFT raises L by 12.55 points and lowers C by 13.32 points, while O rises by 0.77 points. The gain is therefore predominantly a reduction in canonical rewriting, with a small fraction moving into other errors rather than into correct reads. CER falls from 27.77% to 19.24% after SFT.
  • Cost to standard STR. This is real and we report it as a cost. STR36 accuracy is 87.92% for Base and 87.38 / 87.61 / 87.32% after SFT / continued SFT / GRPO. All three Base→SFT paired intervals are below zero. Training on printed misspellings costs a small but consistent amount of ordinary recognition accuracy; we do not describe the post-training as lossless.
  • GRPO. GRPO exceeds continued SFT by +1.07 points in L on average, with per-seed values +0.77 / +2.56 / −0.13 and only seed 1 having an interval above zero, and CER is higher in all three GRPO branches. The matched-budget control does not reproduce a stable advantage. We have revised the claim accordingly.

References

[1] HunyuanOCR-1.5: Making Lightweight OCR VLMs Faster and Better

[2] Scene Text Recognition with Permuted Autoregressive Sequence Models

[3] Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition

[4] An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.