acceptodds
Under review as a conference paper at ICLR 2027

WHEN MLLMS MISS WHAT HUMANS READ: DIAGNOSING AND LEARNING TO READ SQUINT TEXT

Abstract

Multimodal LLMs can fail to read text embedded in an image’s coarse structure— text a human recovers by squinting—and report that no text is present. We name this class Squint Text and study it as a recognition deficiency, motivated by cases from a production moderation pipeline. We introduce SQUINTBENCH, compris- ing 1076 production-derived moderation images, and SQUINTCTRL, a controlled synthetic suite for mechanism diagnosis. Frozen-weight analysis shows that hid- den text leaves a coarse signal while glyph identity is weakly represented; iden- tity becomes substantially more accessible when the correct spatial regions are supplied, implicating spatial selection as a major bottleneck. Suppressing high- frequency content also improves reading at a fixed visual-token budget, while vi- sual token count may also contribute. These findings motivate SQUINT OPSD: a frozen teacher receives a downscaled view and the ground-truth transcription as privileged context, and its token-level distributions supervise a student along on-policy trajectories. At inference, the student reads the original image alone. SQUINT OPSD reaches 88.61 ANLS on SQUINTBENCH, versus 26.50 for the unadapted backbone, 24.44 for the best general-purpose model on the original view, and 68.60 at best for an off-the-shelf model under a downscaled view. These results substantially narrow the gap between available visual evidence and its gen- erated read-out without auxiliary views at inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal large language models, robustness, text recognition, content moderation, knowledge distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_QMgCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies “Squint Text”: text embedded in coarse image structures that humans can recognize by squinting or reducing image scale, but multimodal language models often fail to transcribe. The authors introduce SquintBench, containing 1,076 production-derived images, and SquintCtrl, a synthetic diagnostic suite. They propose SquintOPSD, which distills a frozen teacher conditioned on the original image, a downscaled view, and the ground-truth transcription into a student receiving only the original image. The method achieves substantial improvements over the backbone, SFT, and DAPO, with transfer to external hidden-text benchmarks.

Critical strengths
  • Relevant problem with practical motivation. Hidden-text recognition is a meaningful failure mode for visual content moderation. The combination of production-derived evaluation and controlled synthetic experiments is valuable.
Critical weaknesses
  • The evidence does not yet establish spatial selection as a major causal bottleneck. Oracle pooling uses exact glyph masks that encode shape as well as location, so its advantage over global/random pooling does not isolate localization. Moreover, vision-only adaptation substantially outperforms LM-only adaptation (71.12 vs. 29.25), suggesting an important role for representation changes. Location-only controls and interventions on the frozen model would strengthen this claim.

  • The target-form ablation may confound soft supervision with on-policy training. It is unclear whether SFT-distil and SquintOPSD share identical student-generated prefixes and training settings; thus, the 68.31→88.61 gain cannot be attributed solely to soft targets. A matched hard-versus-soft comparison is needed. The authors should also explain why the mismatched-view experiment uses a paired baseline of 84.60 rather than 88.61.

Questions for authors
  • Does SFT-distil use exactly the same on-policy student prefixes and optimization settings as SquintOPSD? What happens when only the soft teacher distribution is replaced by its argmax target?

  • What configuration differs between the default run and the matched/mismatched-view pair? Can the teacher-input ablations be reported under one consistent setting?

5 comments
AuthorsOpenReview ↗

Response to Reviewer QMgC — Part 1/2

We thank the reviewer for the constructive suggestions. We added controls for spatial pooling and trajectory source, clarified the supervision comparison, and checked the reported matched-view baseline.

W1. Spatial selection and visual representations

The evidence does not yet establish spatial selection as a major causal bottleneck. Oracle pooling uses exact glyph masks that encode shape as well as location, so its advantage over global/random pooling does not isolate localization. Moreover, vision-only adaptation substantially outperforms LM-only adaptation (71.12 vs. 29.25), suggesting an important role for representation changes. Location-only controls and interventions on the frozen model would strengthen this claim.

Response to W1.

Thank you for this suggestion. We agree that the original oracle pooling uses an exact glyph mask, so its advantage may reflect both text location and stroke shape. To further examine their roles, we added two pooling controls on features from the frozen visual encoder and used a linear probe for character recognition.

  1. Preserve mask shape while changing its location. We randomly translate the complete glyph mask, preserving its shape and selected-token count. Each image uses five distinct nonzero translations, without cropping or wrapping. At L18, the exact mask at its original location achieves a macro-AUC of 0.774, compared with 0.649 after translation.

The two conditions use masks of exactly the same shape, but translation substantially reduces performance. Preserving the selector’s shape alone is therefore insufficient: selecting locations aligned with the text is important for reading out glyph information.

  1. Preserve coarse location without exact stroke shape. Within the coarse window containing the text, we randomly select the same number of tokens as the oracle, with five independent samples per image. At L18, this local-random-K control achieves a macro-AUC of 0.684, higher than global random sampling. Thus, knowing the approximate text location is useful even without following the strokes. The exact mask further improves macro-AUC to 0.774, showing an additional benefit from selecting features in the glyph region.

Together, these controls support spatial selection as an important factor in reading out hidden-text information. We have added the control settings and results in §3.2 and the Figure 2 caption and updated the corresponding statement in the abstract.

  1. Role of visual representations. We agree that visual-representation changes are also important. In the module-adaptation experiment, vision-only adaptation achieves 71.12 ANLS, compared with 29.25 for LM-only adaptation. The new controls demonstrate the effect of spatial selection on frozen-feature readout, while module adaptation shows the importance of changing visual representations. We have added this discussion to “Glyph representations become more discriminative” in §6, Analysis.
AuthorsOpenReview ↗

Response to Reviewer QMgC — Part 2/2

W2. Confounding trajectory source and supervision form

The target-form ablation may confound soft supervision with on-policy training. It is unclear whether SFT-distil and SquintOPSD share identical student-generated prefixes and training settings; thus, the 68.31→88.61 gain cannot be attributed solely to soft targets. A matched hard-versus-soft comparison is needed. The authors should also explain why the mismatched-view experiment uses a paired baseline of 84.60 rather than 88.61.

Response to W2.

  1. Trajectory source and supervision form. Thank you for this suggestion. We agree that the original comparison between SFT-distil and Squint OPSD changes both trajectory source and supervision form; the full gain from 68.31 to 88.61 should not be attributed to soft targets alone. We therefore added a 2×22 \times 2 comparison. All four conditions use the same teacher, with the original image, a 1/8 thumbnail, and the GT hint.
Trajectory sourceHardSoft
Offline: cached teacher trajectories68.3177.57
Online: student-generated trajectories72.6788.61

Within the same trajectory-source setting, online-hard and online-soft achieve 72.67 and 88.61, while offline-hard and offline-soft achieve 68.31 and 77.57. The soft-supervision scheme performs better under both trajectory sources.

The two online conditions each use trajectories generated by their own student; the individual prefixes are not required to be identical. We have added the settings and results to Table 4 and its analysis in §5.3 and revised the interpretation of the original comparison in Appendix C.

  1. Matched-view baseline. Thank you for identifying this inconsistency. We rechecked the experimental records and found that 84.60 was a reporting error in the manuscript; 88.61 is the correct single-checkpoint result. We have corrected the corresponding descriptions in §5.3 and Appendix C.

Q1. Student prefixes and hard-target supervision

Does SFT-distil use exactly the same on-policy student prefixes and optimization settings as SquintOPSD? What happens when only the soft teacher distribution is replaced by its argmax target?

Response to Q1.

Thank you for this question. SFT-distil in the original paper uses trajectories generated offline by the teacher, whereas Squint OPSD uses trajectories generated online by the student. They therefore do not use the same on-policy student prefixes, and the original comparison does not isolate supervision form.

We added an online-hard condition using the same teacher as Squint OPSD, with the original image, a 1/8 thumbnail, and the GT hint. Training uses student-generated online trajectories, with the teacher’s full-vocabulary argmax token at the current prefix as the hard target.

Supervision on student-generated trajectoriesANLS
Hard: teacher argmax target72.67
Soft: teacher distribution88.61

Online-hard achieves 72.67 ANLS, compared with 88.61 for soft supervision. Under our experimental settings, the scheme that retains the teacher distribution performs better.

We have added the online-hard result and experimental settings to Table 4 and its analysis in §5.3.

Q2. Matched-view configuration and the default setting

What configuration differs between the default run and the matched/mismatched-view pair? Can the teacher-input ablations be reported under one consistent setting?

Response to Q2.

Thank you for identifying this inconsistency. We rechecked the experimental records and found that the matched-view baseline of 84.60 was a reporting error; the correct result is 88.61. The matched- and mismatched-view conditions use the same experimental settings and differ only in whether the teacher’s auxiliary view matches the current example. Their ANLS scores are 88.61 and 78.60, respectively, a difference of 10.01 points.

The teacher-input ablations use a consistent setting, varying the information supplied to the teacher. We have corrected the corresponding numbers and descriptions in §5.3 and Appendix C.

Thank you for the additional experiments and clarifications, which have largely addressed my previous concerns. I see the paper’s main contributions as the diagnosis of squint-text recognition failures and a training approach tailored to this problem. Since SquintBench provides the main empirical evidence, its diversity and representativeness are important for assessing the generalizability of the reported gains. Could the authors provide a dataset link and access instructions? If access is restricted, a broader selection of representative images spanning different visual styles, text layouts, and sources, together with a brief description of the sampling and deduplication procedures, would help clarify the benchmark’s coverage.

Reviewer_6c5COpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies Squint Text, a type of hidden text that humans can recognize by squinting or downscaling images but MLLMs often fail to read. The authors introduce SquintBench and SquintCtrl to evaluate this problem and investigate its causes. Their experiments suggest that spatial localization and high frequency interference are important factors. Based on these findings, they propose Squint OPSD, which uses a teacher with privileged visual and label information to improve a student's ability to read the original image. The method achieves 88.61 ANLS on SquintBench, substantially outperforming the baselines.

Critical strengths
  1. Interesting problem and clear motivation. The paper identifies a practical failure mode in MLLMs. The diagnostic experiments on visual representations, spatial localization, and frequency interference provide useful insights into why models struggle with hidden text.
  2. Strong results and detailed analysis. Squint OPSD achieves significant improvements over SFT and DAPO. The ablation studies are relatively comprehensive, and the analysis of visual representations provides additional evidence that the model learns to recognize hidden text more effectively.
Critical weaknesses
  1. Limited methodological novelty. The proposed framework is very similar to Vision-OPD, which already uses privileged visual information and on policy distillation. The main differences appear to be the choice of corrective view and the additional ground truth hint. Furthermore, the paper does not directly compare on policy and off policy distribution distillation under the same teacher setting, making it difficult to identify the contribution of the on policy mechanism.
  2. Limited evidence of generalization. Most qualitative examples contain Chinese text, but the paper does not clearly report the language distribution of SquintBench. Although the synthetic experiments include Latin characters, they cover only a small character vocabulary. The method is also trained and evaluated on a single backbone, leaving its effectiveness across languages and model architectures unclear.
Questions for authors
  1. Could the authors clarify the main methodological differences between Squint OPSD and Vision-OPD? Could an off policy distribution distillation baseline using the same teacher and JSD objective be included to better demonstrate the contribution of on policy training?
  2. Could the authors clarify the language distribution of SquintBench and provide results on English hidden text? This would help assess whether the method generalizes beyond the types of text shown in the paper.
  3. How was the human readability of SquintBench verified? Were the transcriptions checked by multiple annotators, and could the authors provide human recognition accuracy or annotation agreement statistics?
  4. The unadapted Qwen3.5-9B outperforms several much larger MLLMs on SquintBench. Could the authors explain this result and clarify whether language distribution or differences in image preprocessing might affect the comparison?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 6c5C — Part 1/3

We thank the reviewer for the helpful comments on methodological contribution and generalization. We clarify the relationship to Vision-OPD and report the additional controlled comparisons and character-level analyses below.

W1. Methodological contribution and the on-policy mechanism

Limited methodological novelty. The proposed framework is very similar to Vision-OPD, which already uses privileged visual information and on policy distillation. The main differences appear to be the choice of corrective view and the additional ground truth hint. Furthermore, the paper does not directly compare on policy and off policy distribution distillation under the same teacher setting, making it difficult to identify the contribution of the on policy mechanism.

Response to W1.

Our initial goal was to extend on-policy self-distillation (OPSD) to image tasks, using additional information available to the teacher to improve recognition of hidden text in AIGC images. During this research, we identified Vision-OPD as related work that also explores on-policy distillation with privileged visual information and discussed it in the paper. The two works differ in their research problem and teacher-input design.

  1. Research problem. Vision-OPD studies fine-grained visual understanding, helping models extract local details needed to answer questions about an image. We specifically study squint-text recognition: character strokes are blended with generated backgrounds, so the model must both locate the text and distinguish and transcribe its glyphs amid complex textures. To our knowledge, this is the first work to improve recognition of this type of hidden text through post-training, and the first application of OPSD to this problem, motivated by real-world content moderation.

  2. Corrective information for the teacher. Vision-OPD provides enlarged local regions to reveal fine-grained details. For hidden text blended with background textures, we instead provide a thumbnail of the entire image. Downsampling suppresses distracting texture and makes the text contours easier to recognize. We also supply the ground-truth transcription as a textual hint. The student still uses only the original image at inference. Our ablations show that combining the thumbnail and textual hint is substantially more effective than using either alone.

  3. Broader research contribution. Beyond an improved recognition method, we introduce a real-world benchmark, systematically diagnose the difficulties of recognizing hidden text in AIGC images, and analyze the role of different visual views. This is the first work to jointly address benchmark construction, diagnostic analysis, and method development for this problem.

  4. On-policy versus off-policy distribution distillation. To examine the on-policy mechanism, we added a 2×22 \times 2 comparison. All four conditions use the same teacher, which receives the original image, a 1/8 thumbnail, and the GT hint. We compare trajectories generated online by the student with trajectories generated and cached offline by the teacher, under both hard and soft supervision. The offline-hard condition corresponds to SFT-distil in the original ablation.

Trajectory sourceHardSoft
Offline: cached teacher trajectories68.3177.57
Online: student-generated trajectories72.6788.61

Under soft distribution distillation, student-generated trajectories improve ANLS by 11.04 points over cached teacher trajectories. Under hard supervision, the gain is 4.36 points. These results demonstrate the benefit of training on the student’s own trajectories under the same teacher setting.

Soft supervision also outperforms hard supervision under both trajectory sources.

We have added the experimental settings and results to Table 4 and its analysis in §5.3.

AuthorsOpenReview ↗

Response to Reviewer 6c5C — Part 2/3

W2. Language and model generalization

Limited evidence of generalization. Most qualitative examples contain Chinese text, but the paper does not clearly report the language distribution of SquintBench. Although the synthetic experiments include Latin characters, they cover only a small character vocabulary. The method is also trained and evaluated on a single backbone, leaving its effectiveness across languages and model architectures unclear.

Response to W2.

Language coverage. SquintBench includes substantial mixed-script content: 1,035 of its 1,076 images (96.19%) contain both Chinese characters and Latin letters. These mixed images contain 4,588 Chinese characters and 4,303 Latin letters, a ratio close to 1:1. The benchmark therefore evaluates both character types rather than Chinese characters alone.

On the full SquintBench dataset, Squint OPSD achieves character-projection ANLS scores of 84.09 for Chinese characters and 94.20 for Latin letters, compared with 70.94 and 85.16 for SFT. We have added the character composition in §2.2, the main findings in §5.2, and the protocol and full results in Appendix B.2. See our response to Q2 below for the detailed comparison.

Cross-architecture evaluation. We additionally compare Base, SFT, and Squint OPSD on InternVL 3.5-8B and Ministral 3-3B.

ModelBaseSFTOPSDΔ\Delta (OPSD − SFT)
InternVL 3.5-8B12.0760.8773.37+12.50
Ministral 3-3B6.8848.1112.51−35.60

Scores are ANLS (%); Δ is the change from SFT in percentage points. On InternVL 3.5-8B, Squint OPSD improves over SFT by 12.50 points, showing that the gain extends beyond our main backbone. On Ministral 3-3B, however, OPSD underperforms SFT. Inspection of training samples shows that its teacher achieves a mean ANLS of only 30.56 despite receiving the original image, a thumbnail, and the correct transcription. For example, even when the hint explicitly provides “Megan Kelley Hall”, the teacher generates a book description and a list of authors instead. This indicates unstable adherence to the answer hint in this setting, which can introduce off-task supervision and contribute to the poor distillation result. The contrasting outcomes also highlight the importance of teacher supervision quality.

We have added the full cross-model results and the Ministral 3-3B teacher diagnostic in Appendix H, with a pointer in §5.2.

Q1. Differences from Vision-OPD and the off-policy baseline

Could the authors clarify the main methodological differences between Squint OPSD and Vision-OPD? Could an off policy distribution distillation baseline using the same teacher and JSD objective be included to better demonstrate the contribution of on policy training?

Response to Q1.

  1. Differences from Vision-OPD. As discussed above, the two methods address different visual-recognition problems. Vision-OPD uses enlarged local views to help the teacher access fine-grained details. For hidden text in AIGC images, we use a thumbnail to make the hidden text more discernible and provide the ground-truth transcription as an additional hint.

Our work is the first to improve recognition of AIGC-generated hidden text through post-training. In addition to the method, we introduce SquintBench and conduct diagnostic experiments on recognition difficulties and the role of different visual views.

  1. Off-policy distribution-distillation baseline. Following the reviewer’s suggestion, we added an off-policy baseline with the same teacher and JSD objective. Both teachers receive the original image, a 1/8 thumbnail, and the GT hint. Off-policy training uses cached teacher trajectories, whereas on-policy training uses trajectories generated online by the student.

Off-policy and on-policy training achieve ANLS scores of 77.57 and 88.61, respectively, a gain of 11.04 points. This comparison shows that training on the student’s own trajectories improves distillation under the same teacher and soft-supervision setting.

We have added this comparison’s settings and results to Table 4 and its analysis in §5.3.

AuthorsOpenReview ↗

Response to Reviewer 6c5C — Part 3/3

Q2. Language distribution and recognition of Latin-letter text

Could the authors clarify the language distribution of SquintBench and provide results on English hidden text? This would help assess whether the method generalizes beyond the types of text shown in the paper.

Response to Q2.

Thank you for this suggestion. We have quantified the character composition of SquintBench and analyzed recognition of Chinese characters and Latin letters.

  1. Character composition. Although many qualitative examples in the paper contain Chinese text, SquintBench contains substantial mixed-script content. Of the 1,076 images, 1,035 (96.19%) contain both Chinese characters and Latin letters, 22 contain Chinese characters but no Latin letters, and 19 contain Latin letters but no Chinese characters.

The 1,035 mixed-script images contain 4,588 Chinese characters and 4,303 Latin letters, a ratio close to 1:1. Thus, our main evaluation covers both character types rather than Chinese characters alone. We have added the mixed-script proportion and character counts in §2.2, Datasets.

  1. Recognition of both character types. On the full SquintBench dataset, we separately extract Chinese characters or Latin letters from the ground truth and predictions and compute character-projection ANLS.
ModelChinese-character ANLSLatin-letter ANLS
BASE24.3433.46
SFT70.9485.16
Squint OPSD84.0994.20

Squint OPSD outperforms both BASE and SFT on both character types. Relative to SFT, it improves Chinese-character and Latin-letter ANLS by 13.15 and 9.04 points, respectively, demonstrating improvements in recognition of both.

This analysis projects the characters appearing in the images; much of the text consists of URLs and promotional text. We have added the main results to the “Target performance” paragraph of §5.2, Main Results, and the full table and scoring protocol to Appendix B.2.

Q3. Verification of human readability

How was the human readability of SquintBench verified? Were the transcriptions checked by multiple annotators, and could the authors provide human recognition accuracy or annotation agreement statistics?

Response to Q3.

Thank you for this suggestion. SquintBench contains hidden-text examples that human reviewers could recognize in real-world content moderation. Human annotators transcribed the text, and multiple annotators checked the transcriptions.

To quantify readability, we invited six participants to transcribe all 1,076 SquintBench images, obtaining an ANLS of 98.6%. Remaining errors mainly involve visually similar characters in URLs and other strings, such as i/l and c/o, which can be difficult to distinguish after AIGC-based hiding.

We have added the human-recognition result in §2.2, Datasets, replacing “almost effortlessly” with this quantitative measurement.

Q4. Qwen versus larger models

The unadapted Qwen3.5-9B outperforms several much larger MLLMs on SquintBench. Could the authors explain this result and clarify whether language distribution or differences in image preprocessing might affect the comparison?

Response to Q4.

Thank you for this suggestion. We further analyzed the effects of character composition and image preprocessing on the comparison.

  1. Recognition of different character types. SquintBench contains 1,035 images with both Chinese characters and Latin letters. We separately extract the two character types from the ground truth and model outputs and compute ANLS.
ModelChinese-character ANLSLatin-letter ANLS
Qwen3.5-9B BASE24.2733.49
Claude14.2833.48
Gemini15.1429.00
GPT-astra7.9420.90

Compared with Claude, Qwen scores substantially higher on Chinese characters, while their Latin-letter scores are nearly identical. Stronger Chinese-character recognition may therefore be one reason for Qwen’s better performance on SquintBench.

We have added this observation to the “External transfer” paragraph of §5.2, Main Results, and the full results to Appendix B.2.

  1. Image preprocessing. Corresponding experiments use the same original-image or thumbnail lists. Thumbnails are generated with BICUBIC resizing and JPEG quality 95. Qwen’s local processor rescales images that are too large or too small to its supported pixel range. Commercial models process images through their respective APIs, whose internal resizing and visual-token allocation may differ. These preprocessing differences may also affect the comparison.
Reviewer_h49iOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a view on an important problem of visual inability to detect coarse texts in images. MLLMs often fail to read hidden text within the coarse structure of images, which is often a means of broadcasting harmful and inappropriate information as reported by Tiktok, Meta and other social media based companies. Usually these texts can be read by people by focussing carefully/ squinting on the image, yet MLLMs deny the presence of text in such images. The authors build a benchmark called SquintBench consisting of 1076 production moderation images consisting of chinese glyphs. Moreover they also contribute SquintCtrl which is a synthetic suite where the glyph mask is known exactly. They find on Qwen3.5-9B that the presence of hidden text is encoded good enough but glyph identity itself is weaker. Although it becomes stronger when the features are pooled at glyph locations. They pose an on-policy self distillation method wherein a frozen teacher sees the original image input, a 1/8 downscaled view and the ground truth description. This model supervises a student model which only sees the input image.

Critical strengths
  1. I liked the real world relevance to the problem, since content moderation is quite necessary in the present day and age. The pairing of a realistic benchmark with a controlled synthetic suite is a good design.

  2. The diagnosis uses sensible controls. They compare oracle, global and random pooling. They test frequency at constant token count, add contrast-only controls such as CLAHE, which gives no gain, and check what the patch merger changes. They are also honest that token count and frequency cannot be fully separated.

Critical weaknesses
  1. I feel the training and test data could potentially be overlapping or too close. The training mix includes 3,489 synthetic hidden-text images from an AIGC collection, FakeTextDetect, which arent exactly cited or properly documented. The authors do not say whether SquintBench was deduplicated against the training data, or whether SquintCtrl-Read and SquintCtrl-Glyph, which are used for the post-training analysis, are disjoint from the ControlNet training images made with the same pipeline.

  2. The authors do not quantify the human-legibility for SquintBench. The paper says humans read the text “almost effortlessly,” yet human readers recovered only 35.7% of HC-Bench. No legibility rate or inter-annotator agreement is reported for SquintBench itself.

Questions for authors

I had a few questions for the authors, I would like answered

  1. For the commercial models, were refusals or safety-filtered responses counted as failures? The content is promotional and policy-violating, and the very low gpt-6-astra scores even on the downscaled views could suggest something other than reading failure.

  2. Is SquintBench deduplicated against the training data, including near-duplicates from the same generators? What are the AIGC collection and FakeTextDetect sources?

  3. The hint-only teacher performs far below SFT on the same ground truth, based on Table 4, even though that teacher is given the correct transcription. Why does distilling from a teacher that knows the answer work so much worse than training on the answer directly? How often does the teacher’s top token match the ground truth under each teacher condition?

5 comments
AuthorsOpenReview ↗

Response to Reviewer h49i — Part 1/3

We thank the reviewer for the constructive comments. Below, we clarify the data sources and evaluation procedures and report the additional teacher diagnostics.

W1. Training-data sources and dataset independence

I feel the training and test data could potentially be overlapping or too close. The training mix includes 3,489 synthetic hidden-text images from an AIGC collection, FakeTextDetect, which arent exactly cited or properly documented. The authors do not say whether SquintBench was deduplicated against the training data, or whether SquintCtrl-Read and SquintCtrl-Glyph, which are used for the post-training analysis, are disjoint from the ControlNet training images made with the same pipeline.

Response to W1.

Thank you for pointing out that the training-data sources and dataset splits were not sufficiently clear. SquintBench, SquintCtrl-Read, and SquintCtrl-Glyph were constructed independently of the training data, with no overlapping samples between training and evaluation. We have clarified their sources and construction as follows.

  1. Composition of the AIGC-related training data. The 3,489 examples are not all positive examples of synthetic hidden text. Before filtering, AIGC collection contained 1,143 examples, self-synthesised ControlNet contained 1,144, and FakeTextDetect contained 1,267. All FakeTextDetect examples have [] as their target answer and serve as no-text negatives. We removed 65 low-quality examples from the combined collection, leaving 3,489 examples.

AIGC collection uses QR Monster, primarily with SD 1.5 as the diffusion model; self-synthesised ControlNet uses QR Monster 2. The training data have been released: https://anonymous-hf.com/a/qjh55c6odaz5/. We have revised the training-data table in Appendix F.1 and added the purpose and generation procedure of each source.

  1. Relationship between SquintBench and the training data. SquintBench consists of real-world content-moderation images originating from illicit promotional activity; their generation procedures are unknown. In contrast, the synthetic training images use random text strings that do not overlap with the text strings in SquintBench. They use single-line or multiline layouts and background prompts sampled from JourneyDB. The training and evaluation data are therefore separated in both their text content and background sources. We have clarified their separate construction in §2.2 and described the synthetic training-data generation process in Appendix F.1.

  2. Relationship between SquintCtrl and the training data. Although both use ControlNet-based hidden-text generation, their text, conditioning images, and background pools are constructed separately. The training images use a broader range of text and single-line or multiline layouts. SquintCtrl-Glyph instead combines 2–3 characters sampled from fixed Chinese and Latin character sets and uses separately constructed conditioning images. The two collections do not share a background pool, and their images do not overlap. We have added these construction differences in Appendix F.1.

SquintBench is publicly available: https://anonymous-hf.com/a/jci2bjb3m7x2/.

W2. Human readability

The authors do not quantify the human-legibility for SquintBench. The paper says humans read the text “almost effortlessly,” yet human readers recovered only 35.7% of HC-Bench. No legibility rate or inter-annotator agreement is reported for SquintBench itself.

Response to W2.

Thank you for pointing out the lack of quantitative evidence for human readability. We agree that the original phrase “almost effortlessly” was imprecise.

SquintBench was collected from real-world content-moderation scenarios. Human annotators transcribed the text, and multiple annotators checked the transcriptions. To quantify human readability, we invited six participants to transcribe all 1,076 images, obtaining an ANLS of 98.6%. Remaining errors include visually similar characters in URLs, such as i/l and c/o.

We have added the multiple-annotator checking procedure and human-recognition result in §2.2, replacing “almost effortlessly” with this quantitative measurement.

AuthorsOpenReview ↗

Response to Reviewer h49i — Part 2/3

Q1. Refusals and safety filtering

For the commercial models, were refusals or safety-filtered responses counted as failures? The content is promotional and policy-violating, and the very low gpt-6-astra scores even on the downscaled views could suggest something other than reading failure.

Response to Q1.

Thank you for this question. We inspected the commercial models’ responses under the original-image and two downsampled-view conditions and categorized their failure outputs.

  1. Explicit refusals were included in the evaluation. We found one explicit refusal among GPT-astra’s original-image responses: it interpreted the image as a CAPTCHA and declined to recognize it. This response was included and received a score of zero. We found no explicit safety refusals in GPT-astra’s two downsampled conditions, or explicit refusals concerning promotional or policy-violating content in the Claude and Gemini records.

  2. GPT-astra’s low scores are associated primarily with frequent responses stating that the image cannot be read. The table below summarizes the failure outputs.

ModelViewTotal failuresNonempty, ANLS = 0Cannot-read/view explanationEmptyOther
GPT-astraOriginal947365561192
GPT-astra1/894724470111
GPT-astra40×4097422375001
ClaudeOriginal74430704370
Claude1/8745227594590
Claude40×408816071121620
GeminiOriginal81637104450
Gemini1/866819004780
Gemini40×403503290210

GPT-astra produced 561, 701, and 750 cannot-read/cannot-view explanations under the three view conditions, respectively, substantially more than Claude or Gemini. Its low scores thus accompany many responses that do not complete the transcription, rather than many explicit safety refusals.

Claude and Gemini exhibit different failure patterns. With original images, they returned 437 and 445 empty results, respectively. At 40×40, a larger share of their failures consisted of nonempty transcriptions with ANLS = 0.

Q2. Deduplication and data sources

Is SquintBench deduplicated against the training data, including near-duplicates from the same generators? What are the AIGC collection and FakeTextDetect sources?

Response to Q2.

Thank you for this question. SquintBench contains real-world content-moderation images and was constructed separately from the synthetic training data, with no overlapping samples. The synthetic training text consists of random strings whose full text strings do not overlap with SquintBench.

Regarding the training sources, AIGC collection uses QR Monster, primarily with SD 1.5 as the diffusion model. Self-synthesised ControlNet uses QR Monster 2. All 1,267 FakeTextDetect examples use [] as the target and serve as no-text negatives, rather than positive synthetic hidden-text examples.

We have clarified dataset construction, training-data sources, and the purpose of each source in §2.2 and Appendix F.1.

AuthorsOpenReview ↗

Response to Reviewer h49i — Part 3/3

Q3. Why does the hint-only teacher underperform direct SFT?

The hint-only teacher performs far below SFT on the same ground truth, based on Table 4, even though that teacher is given the correct transcription. Why does distilling from a teacher that knows the answer work so much worse than training on the answer directly? How often does the teacher’s top token match the ground truth under each teacher condition?

Response to Q3.

Thank you for this question. To understand why distillation from the hint-only teacher underperforms SFT despite receiving the correct transcription, we examined teacher predictions under both ground-truth prefixes and student-generated prefixes.

  1. Token prediction under ground-truth prefixes. Using the same ground-truth prefixes, we measured full-vocabulary top-1 accuracy over 35,307 transcription-text tokens.
Teacher inputGT top-1 accuracy
Original + hint98.13%
Original + thumbnail76.18%
Original + thumbnail + hint95.95%

The hint-only teacher accurately predicts characters given correct text prefixes. However, these statistics begin within the text field and do not include the earlier decision of whether to output text at all.

  1. Prediction immediately before the student chooses an empty answer. We sampled outputs from the initial BASE student on 256 images containing text; the student returned an empty answer for 123 of them. For these cases, we gave the three teachers the same student prefix immediately before the empty-answer decision and measured how often they ranked the correct nonempty-answer opening token first.
Teacher inputCorrect opening-token top-1 accuracy
Original + hint10.57% (13/123)
Original + thumbnail76.42% (94/123)
Original + thumbnail + hint86.18% (106/123)

Although the hint-only teacher receives the correct transcription, it often also favors an empty answer in these cases. Adding the thumbnail makes the teacher substantially more likely to choose the correct nonempty opening.

SFT directly supervises the correct output, whereas hint-only distillation depends on the teacher’s distribution at student-generated prefixes. When the student tends toward an empty answer, the hint-only teacher provides a weaker corrective signal. This difference explains a concrete weakness of hint-only supervision relative to direct SFT.

  1. Other error positions. At the 28 positions where the student first made an error on a purely textual token, the hint-only and full teachers achieved GT top-1 accuracies of 67.86% and 71.43%, respectively. They also showed no clear difference in continuation after an error. The more pronounced distinction in these diagnostics is therefore the empty-answer decision.

We have summarized the teacher diagnostics in §5.3 and added the full token-accuracy, empty-answer-decision, and error-continuation results in Appendix C.1.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Summary of Revisions

We sincerely thank the reviewers and Area Chair for their careful evaluation and constructive suggestions. We have revised the manuscript with additional experiments, analyses, and clarifications. Below, we summarize our core contributions and the main changes made during rebuttal. Detailed responses follow under each review.

Core Contributions of Our Work

  1. AIGC hidden-text recognition. We study the recognition of text embedded in AIGC images, combining a real-world benchmark, controlled diagnostic experiments, and a post-training solution for this content-moderation challenge.
  2. Diagnosis of recognition difficulties. Our analyses examine spatial selection, visual representations, and image-scale effects, providing evidence for why hidden text can be readable to humans yet difficult for MLLMs to transcribe.
  3. Single-image recognition enhancement. Squint OPSD uses a thumbnail view and a ground-truth hint to provide privileged teacher supervision during training, while the student uses only the original image at inference.

Additional Experiments during Rebuttal

  • §2.2: Added a human transcription evaluation with six participants on all 1,076 SquintBench images, obtaining 98.6% ANLS.
  • §3.2 and Figure 2: Added translated-mask and local-random-token controls on frozen visual features to examine spatial selection beyond the original exact-glyph-mask pooling experiment.
  • §5.3 and Table 4: Added a 2×22 \times 2 comparison of on-policy/off-policy trajectories and hard/soft supervision under the same teacher-input setting.
  • §5.3 and Appendix C.1: Added teacher-token diagnostics under gold and student-generated prefixes, including empty-answer decisions, first text-token errors, and continuation after errors.
  • §5.2 and Appendix B.2: Added Chinese-character and Latin-letter transcription results, together with a script-specific comparison of Qwen and commercial models.
  • Appendix H, referenced in §5.2: Added Base/SFT/OPSD comparisons on InternVL 3.5-8B and Ministral 3-3B.

Clarifications and Discussion Added during Rebuttal

  • §2.2 and Appendix F.1: Clarified training-data sources, no-text negative examples, and the separate construction of training and evaluation data.
  • §2.2 and Appendix B.2: Reported the character composition of SquintBench and specified the character-projection evaluation protocol.
  • §3.2 and §6: Clarified spatial selection as an important factor and discussed the complementary role of visual representation changes.
  • §5.3 and Appendix C: Clarified the trajectory sources and supervision settings, and corrected the matched-view baseline to 88.61.
  • §5.2 and Appendix B.2: Discussed language composition and model-specific preprocessing when interpreting comparisons with larger MLLMs.

The main additions are highlighted in blue in the revised manuscript. Each detailed response identifies the corresponding manuscript changes and locations. We thank the reviewers again for helping us improve the clarity and empirical support of our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.