- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces SceneFaith, a benchmark of 781 generated scene images across 17 categories in which an unusual printed string conflicts with a contextually plausible conventional spelling. Outputs are classified as Literal, Canonical, or Other, and 15 model variants from seven families are evaluated. The main diagnostic contribution is to compare predictions after removing peripheral content and after changing the scene around an identical target patch in a separate 150-pair set. Additional experiments examine blur, lexical preference, model variants, consensus filtering, and a post-training pilot. These experiments document canonical substitutions and peripheral-input sensitivity on a deliberately constructed stress test.
Critical strengths
-
The fixed-target-patch interventions are a useful addition to transcription-faithfulness evaluation. Holding the target pixels, position, scale, and local background fixed addresses an important limitation of comparing full images with enlarged crops, and provides evidence that peripheral input affects transcription.
-
The Literal/Canonical/Other breakdown distinguishes recovery of the printed string from changes between different error types. The broad model panel and paired uncertainty estimates make the reported failure patterns informative. The consensus-filtering example also illustrates a practically relevant limitation of agreement-based validation.
Critical weaknesses
-
The information gained beyond existing rewriting studies is limited. As the paper acknowledges, TextHalu-Bench and FaithC4 already document semantically plausible substitutions of unusual text. Confirming such substitutions, together with blur sensitivity and associations with lexical preference, provides limited incremental insight on its own. Holding the target fixed is a meaningful extension, but the conditions governing the observed context sensitivity remain unclear. The increase in rewriting after gray masking in Table 17 is potentially more informative. This uses a different target set from the main benchmark, so it is not a contradiction; however, the conditions governing the change in direction remain uncharacterized. Comparisons with OCR-specialized recognizers on the same targets would also help establish the test's distinctive diagnostic value. Overall, I find the stress test useful, but its demonstrated information gain and evaluation utility insufficient for acceptance.
-
The validity of the intended context manipulation needs clarification. Section 3.2 states that the canonical spelling must not appear elsewhere in an image, yet Figure 2 contains "leopard" in the paragraph outside the box containing "ieopard." I cannot determine whether this example belongs to the evaluated 781 images. If it does, copying another visible word is a competing explanation, and gray masking removes this answer source along with scene content. This is a concrete concern about the stated screening procedure, although it does not establish dataset-wide leakage. Moreover, the A/B comparison changes identifier role and visual appearance alongside scene meaning, as Appendix H acknowledges. The experiments therefore establish peripheral-input sensitivity more securely than a specific semantic-context effect. This limits the interpretation of the paper's main extension beyond prior rewriting studies.
Questions for authors
-
Is the Figure 2 example included in the 781-image benchmark? How was the rule excluding canonical spellings elsewhere in the image implemented, what fraction of evaluated images contain such text, and do the principal full-versus-gray effects remain on a verified leakage-free subset?
-
Can the authors use the existing paired controls to characterize the conditions under which gray masking increases versus decreases rewriting? What evidence distinguishes scene meaning from identifier role, peripheral text, and other visual changes? This would clarify what new understanding the benchmark adds beyond documenting peripheral-input sensitivity.
-
Please clarify effective sample counts and scoring. Table 1's InternVL3-38B blur percentages (26.15/60.51/13.33) are consistent with counts of 204/472/104 out of 780, but not integer counts out of the stated 781; its clear RR also changes from 58.51 in Table 1 to 58.46 in Table 15. Were responses excluded, and which common subset was used for paired analyses? Also, what deterministic rule implements "deletes unrelated text," particularly when an answer includes both the printed and canonical strings?
7 comments
Response to Reviewer YWqz
Q1. Is Figure 2 in the benchmark, how is the no-leak rule enforced, what fraction of images contain such text, and does the main effect survive excluding them?
This question has four parts; we answer each.
- Figure 2 is in the benchmark (
full500_004_animal_A), with twoleopardinstances outside the box. It should not have passed screening, and we say so in the paper rather than replacing the figure silently. - Prevalence. 22 of 781 images (2.82%) contain the exact canonical spelling outside the box; 9 contain only simple inflections; 6 are uncertain. The full-positive upper bound is 37/781 (4.74%).
- Enforcement. The original cohort's scene screening by Gemini 3.1 Flash Lite treated complete canonical words, case variants and simple inflections as exclusion criteria; the crop blind read checked transcription of the target only. The expansion added a GPT-5.5 full-image check for occurrences of either spelling outside the box, with AI-assisted review. The miss in the original cohort shows the check was not exhaustive — the model reported no outside spelling for this image. For this revision we ran a full re-screen (model tiling at native resolution plus Apple Vision OCR with language correction disabled, with visual confirmation of every candidate) rather than a spot check, and report the result.
- Subset recomputation. On the 744-screen-negative set, the Gray − Full RR difference remains negative for all four models with intervals excluding zero (values above). The sensitivity variant that also excludes the uncertain cases (753 pairs) leaves the differences essentially unchanged (−18.19, −5.98, −9.30, −20.05).
Q2. When does masking increase rewriting, and can semantics, identifier role, peripheral text and visual changes be separated?
When the mask increases rewriting. On the 150-patch set, for both Qwen3-VL Instruct aliases (8B and 235B-A22B) masking produces markedly higher RR than either full-scene condition — for 8B Instruct, 25.33% on gray against 8.00% aligned and 4.00% identifier. For Qwen3-VL-32B Thinking, masking lowers RR relative to the aligned scene (8.00% vs 22.00%). On the 781-image main set, masking lowers RR for all four models.
Separation of factors. What the current controls can separate: the target patch's contribution, which is held fixed in both the mask experiment and the A/B experiment, and the joint contribution of the periphery, which is measured by both. What they cannot separate: the individual contributions of scene meaning, word-versus-identifier role, nearby text and low-level appearance. Each intervention changes at least three of these at once. The same-domain shuffle narrows the gap for scene meaning but not for appearance. We do not claim to identify semantics, and the paper states that an appearance-matched, role-matched semantic control would be required to do so, which the current image set does not support. This is a real limitation of the diagnostic design.
Q3. Denominators, pairing subsets, and the parsing rule
Denominators. The provider API returned HTTP 400 for one moderately blurred input, nataexp_0138. InternVL3-38B's Table 1 moderate-blur counts are therefore 204 L / 472 C / 104 O, out of 780. No other responses were excluded from Table 1.
Table 1 vs Table 15. Table 15 pairs clear and blurred responses by image ID and uses the 780 IDs with valid responses in both conditions. For InternVL3-38B the clear RR is 457/781 = 58.51% in Table 1 and 456/780 = 58.46% in Table 15. The difference is the single missing ID, not a different sample.
Parsing rule (deterministic, now in §4.1). Remove an enclosing code fence; keep the first nonempty line; remove a leading Answer:, Output:, Text: or Result: label (case-insensitive) and one pair of matching outer quotation marks or backticks; fold case and normalize whitespace. The entire retained line must match the printed form (L) or the canonical form (C); otherwise O. Both spellings on the retained line → O. Both spellings on separate lines → only the first nonempty line is scored. We also report, in Appendix M, all 15 aliases rescored under strict whole-answer matching: 92/11,715 clear and 60/11,714 moderate answers change label; the moderate RR ranking is unchanged, the clear one has one adjacent swap and one resolved tie; all eight Full − Gray RR differences are identical under both rules.
References
[1] HunyuanOCR-1.5: Making Lightweight OCR VLMs Faster and Better
[2] Scene Text Recognition with Permuted Autoregressive Sequence Models
[3] Read Like Humans: Autonomous, Bidirectional and Iterative Language Modeling for Scene Text Recognition
[4] An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition
Thank you for the response and for the new 2B study. I still have two concerns.
-
Section 4.6 says the 4B pilot trains on 300 clear and blurred pairs and is evaluated on 781 test pairs, which is the whole benchmark. If the 300 pairs come from SceneFaith, they must overlap the 781 evaluated images, since 300 plus 781 is larger than 781. Where do the 300 pairs come from? Are they separate from the 781 images?
-
In your 2B table, continued SFT reaches a CER of 16.59 and GRPO reaches 20.35, so extra steps alone do more than GRPO, and GRPO is worse than SFT on CER, 20.35 against 19.24. However, Section 4.6 still reports a modest but consistent gain over SFT from a comparison against a shorter trained arm.
Response to Reviewer YWqz
We thank the reviewer for their careful reading and for raising these thoughtful questions. We have conducted additional experiments to answer each of them.
W1. Incremental knowledge, the Table 17 reversal, and same-target OCR
What is not new. TextHalu-Bench and FaithC4 already document the replacement of unusual strings by plausible spellings; FaithC4 in particular separates recovery of the unperturbed word from other errors. §2.2 now describes this accurately, and we no longer present the phenomenon as a discovery.
What the interventions add. The fixed-patch diagnostic holds the target patch, including its local background, at identical pixels, position and scale while only the periphery changes. Across 11 aliases, mean RR is 6.85% in identifier scenes and 11.15% in scenes supporting the conventional word: +4.30 points, with 9/11 positive estimates and 5/11 pointwise 95% intervals above zero. This is a paired estimate of peripheral-input sensitivity with a stated per-model spread, not a further demonstration that rewriting occurs.
The Table 17 reversal. We report the reversal with intervals instead of explaining it away. On the 150-patch set, gray-minus-aligned RR differences are +17.33 points [10.67, 24.67] for Qwen3-VL-8B Instruct, +20.00 [12.00, 28.00] for Qwen3-VL-235B-A22B Instruct, and −14.00 [−20.67, −7.33] for Qwen3-VL-32B Thinking. On the 781-image main set, masking lowers RR for all four models tested (6.53–20.10 points clear, 7.68–25.86 under blur). The sign therefore depends on the model, even on a shared set of 150 targets.
We do not have evidence identifying why the sign flips. The two datasets use different targets and a different rendering of the periphery, and the mask removes semantics, nearby text and clutter together. Consequences for the paper: we no longer treat the mask as a neutral "remove context" operation, we describe masking as a manipulation whose effect on rewriting is model- and stimulus-dependent, and the reversal appears in the results discussion rather than only in an appendix table. We consider this an open question the benchmark raises rather than resolves, and we have marked it as such.
Same-target OCR. HunyuanOCR-1.5[1] on the 150 unique patches: 98.67% L, 1.33% C. On clear crops: 91.68 / 3.46; on moderate-blur crops: 83.99 / 10.24. The three STR systems (PARSeq[2], ABINet[3], CRNN[4]) on clear crops span 24.46–84.25% L and 2.30–7.94% C. Given the target region, a dedicated recognizer recovers the printed string much more often, and it still substitutes the conventional spelling at a low rate. This gives the reference point requested: what a system that is not making scene-level inferences does on these strings.
Contribution, restated. The paper now claims three specific measurements rather than the phenomenon: the paired intervention with effect size and heterogeneity, the divergence between RR ranking and literal-accuracy ranking, and the comparison showing that dedicated recognizers substitute the conventional spelling at lower but nonzero rates.
W2. Validity of the context interventions
Figure 2. We confirm the example is the benchmark image
full500_004_animal_A. The figure's PNG matches the evaluation file by SHA-256, andleopardappears twice in the paragraphs outside the target box. In the original pairing, all four models outputleopardon the full image; with the gray mask, GPT-5.2 still outputsleopardand the other three outputieopard. The reviewer's alternative explanation — direct copying of a word visible elsewhere in the image — is correct for this example.How often this happens. We re-screened all 781 images for the exact canonical spelling outside the target box. It occurs in 22 images (2.82%); a further 9 contain only simple inflections, and 6 are uncertain. Combined with the exact cases, the wide criterion gives 31 images (3.97%), and including all uncertain cases gives 37 (4.74%).
Does the main conclusion survive? On the 744 images that screen negative for exact and inflected canonical spellings, the Gray − Full RR differences are −18.15 (GPT-5.2), −5.65 (Gemini 3.5 Flash), −9.01 (Claude Sonnet 4.6) and −19.89 (Qwen3-VL-32B Thinking) points, with all four paired intervals below zero. On the full 781 these are −18.57, −6.53, −9.60 and −20.10. The shrink is modest and the direction is unchanged, so copying cannot account for the overall masking effect. The leak is real, but small.
Framing. The reviewer is also right that A/B changes scene semantics, target role and visual appearance together, so the experiments identify sensitivity to peripheral input and not a semantics-specific effect. That is now the paper's claim, and the limitation is stated where the result appears and again in the Limitations section.