acceptodds
Under review as a conference paper at ICLR 2027

ArmorOCR: Grounded Adversarial Visual Perception via Observation-Transferred Self-Distillation

Abstract

Large multimodal models (LMMs) have demonstrated strong OCR recognition capabilities, yet remain vulnerable to adversarial visual text that is readable to humans but challenging for models to localize and recognize. Existing OCR benchmarks mainly focus on natural or document-style text, while adversarial OCR evaluations remain limited in scale, task coverage, or region-aware evaluation. In this paper, we formulate adversarial OCR as a grounded OCR perception task and introduce AdvSpot, the first benchmark for grounded adversarial OCR evaluation. AdvSpot comprises 383 images with region-level annotations, spanning 5 primary categories and 13 fine-grained adversarial OCR types. To address this challenge, we propose ArmorOCR, a two-stage training framework for robust adversarial OCR perception. ArmorOCR first acquires missing adversarial OCR perception from privileged transformed observations through On-Policy Self-Distillation (OPSD), and then refines grounded OCR perception through Group Relative Policy Optimization (GRPO) with task-conditioned rewards for localization, recognition, full spotting, and visual question answering (VQA). Experiments on our AdvSpot, other adversarial OCR benchmarks, and general OCR benchmarks demonstrate that ArmorOCR consistently improves adversarial OCR perception while preserving competitive general OCR capability. The benchmark and code will be publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Adversarial OCR, On-policy Self-Distillation, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_FKrHOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes the AdvSpot benchmark and ArmorOCR training framework to study large multimodal models' perception capability for hard-to-recognize and hard-to-locate visual text. AdvSpot contains 383 images with region-level annotations, covering 5 categories and 13 text patterns. ArmorOCR employs two-stage training: first distilling visual cues from transformed views, then optimizing text recognition, region localization, and question answering through task rewards.

Critical strengths
  1. The paper evaluates whether models can localize text regions, not merely whether final answers are correct, providing a more complete view of OCR perception.
  2. The distillation stage leverages visual cues from transformed views, while the reinforcement stage separately optimizes recognition, localization, detection, and region-level QA.
  3. AdvSpot provides text regions, transcriptions, pattern categories, and region-level QA, facilitating analysis of different error types.
  4. Results demonstrate that multiple framework components contribute gains, and performance on other adversarial OCR and general OCR benchmarks is reported.
Critical weaknesses

I have the following concerns:

  1. AdvSpot's overall size is modest, with relatively few samples per fine-grained category, and results may be influenced by sample selection.
  2. Some categories more closely resemble general OCR hard cases, and the boundary with adversarial patterns such as concealment and encoding remains insufficiently clear.
  3. Training data is generated by synthetic pipelines and filtered by specific OCR patterns and base model performance; model behavior on different sources or novel text patterns remains unclear.
  4. ArmorOCR is primarily trained on Qwen3-VL-8B, and whether the training scheme transfers to other model architectures has not been demonstrated.
  5. The method requires synthesizing large volumes of training samples and conducting two-stage training; the paper offers limited discussion of the relationship between training resource investment and performance gains.
Questions for authors

see weakness

1 comment
Reviewer_FKrHOpenReview ↗

review

The authors have adequately addressed my concerns. I am satisfied with their responses and clarifications, and I am happy to maintain my positive rating.

Reviewer_DQQyOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces AdvSpot, a benchmark for grounded adversarial OCR perception comprising 383 images and 389 region-grounded VQA pairs across 13 fine-grained visual text patterns. The authors also propose ArmorOCR, a two-stage training framework that first transfers perception from privileged transformed observations through observation-transferred on-policy self-distillation (OTSD), then refines localization, recognition, full spotting, and grounded VQA through task-conditioned GRPO rewards. Experiments compare ArmorOCR with existing LMMs and specialized OCR approaches on AdvSpot, AdvOCR, SmuggleBench, and general OCR benchmarks. The authors report substantial improvements over the Qwen3-VL-8B backbone on AdvSpot (29.8% to 57.1% accuracy) and AdvOCR (21.0% to 56.0%), while largely preserving general OCR performance. Additional ablations examine the contributions of the distillation strategy and individual reward objectives.

Critical strengths
  1. A useful extension of adversarial OCR evaluation: AdvSpot provides region-level annotations, grounded questions, and a relatively systematic taxonomy of difficult visual text patterns. These annotations enable more fine-grained analysis of localization and recognition failures than recognition-only benchmarks.
  2. Promising empirical improvements with meaningful ablations: ArmorOCR substantially improves adversarial OCR performance over its backbone, with additional improvements on external benchmarks. The component and reward ablations provide useful evidence that both privileged visual observations and multi-task refinement contribute to the reported results.
Critical weaknesses
  1. Limited novelty and insufficiently established incremental value:
    • Benchmark contribution: Many of the 13 adversarial OCR subtypes overlap with visual obfuscation patterns already studied in existing benchmarks such as AdvOCR and SmuggleBench [1, 2]. AdvSpot adds finer categorization and region-level annotations, but the paper does not sufficiently demonstrate that it reveals model capabilities or failure modes that existing benchmarks cannot distinguish. With only 383 images and approximately 30–35 examples per subtype, the robustness of fine-grained comparisons is also unclear. Moreover, recognition accuracy and localization IoU are evaluated separately, while the primary VQA metric only requires the ground-truth text to appear as a substring of the response. The evaluation therefore does not directly establish joint recognition-and-localization correctness, nor does it distinguish adversarially induced localization failures from models' general spatial grounding limitations.
    • Training methodology: The proposed framework combines privileged-observation on-policy distillation with task-conditioned GRPO, both building on established techniques. In particular, prior work has already explored transferring privileged visual observations to a full-image student through on-policy distillation [3], while visual augmentation and reinforcement learning for challenging OCR-related perception have also been investigated [1]. The incremental contribution of OTSD remains somewhat unclear because the privileged teacher receives not only transformed observations but also the ground-truth OCR transcription. Consequently, the gains may partially reflect stronger answer supervision rather than uniquely improved visual perception transfer.
  2. Insufficient evidence of generalization and end-to-end methodological advantages:
    • Limited robustness validation: Although ArmorOCR achieves substantial improvements on AdvSpot and the synthetic split of AdvOCR, its performance is less convincing on other distributions. On AdvOCR, it achieves 68% on synthetic samples but only 44% on real-world samples, compared with 48% and 62%, respectively, for VACoT [1]. On the Perceptual Blindness subset of SmuggleBench, the improvement over the backbone is much smaller (13.3% to 17.1% TER), despite the pronounced gains on related visual patterns in AdvSpot. These benchmarks differ in their tasks and metrics, so the results are not directly interchangeable; nevertheless, they raise questions about generalization. The paper does not evaluate held-out image generators, unseen dot/line encoding configurations, or systematically controlled perturbation types and strengths. Since training data are synthetically constructed using related adversarial visual mechanisms, it remains unclear whether the model learns transferable OCR capabilities or primarily adapts to particular synthesis patterns.
    • Missing end-to-end training comparisons: The reported SFT, OPSD, and GRPO baselines primarily compare individual Stage 1 training procedures, while the complete ArmorOCR pipeline additionally uses 70K samples and a substantial Stage 2 GRPO training process. The current ablations are informative, but do not fully establish the advantage of the proposed two-stage design over simpler complete pipelines under comparable supervision and computational budgets. In particular, standard ground-truth SFT followed by the same multi-task GRPO, or supervised distillation using privileged transformed views, would be important alternatives for determining whether the proposed OTSD mechanism is necessary.

[1] Xu et al., VACoT: Rethinking Visual Data Augmentation with VLMs. CVPR Findings 2026. https://openaccess.thecvf.com/content/CVPR2026F/html/Xu_VACoT_Rethinking_Visual_Data_Augmentation_with_VLMs_CVPRF_2026_paper.html

[2] Li et al., Making MLLMs Blind: Adversarial Smuggling Attacks in MLLM Content Moderation. Findings of ACL 2026. https://aclanthology.org/2026.findings-acl.1006/

[3] Yuan et al., Vision-OPD: Learning to See Fine Details for Multimodal LLMs via On-Policy Self-Distillation. arXiv, 2026. https://arxiv.org/abs/2605.18740

Questions for authors
  1. Incremental value of AdvSpot: Beyond offering more fine-grained annotations, what evidence demonstrates that AdvSpot captures capabilities or failure modes not already revealed by existing adversarial OCR benchmarks? Could the authors provide a cross-benchmark analysis of model rankings or distinctive failure cases? A joint recognition-and-localization metric, together with localization performance on comparable non-adversarial text images, would also help establish the value of grounded evaluation.
  2. Generalization to unseen adversarial patterns: How well does ArmorOCR generalize to visual encoding mechanisms or image-generation configurations not represented during training? For example, could the authors evaluate held-out diffusion generators, dot/line pattern configurations, or perturbation strength ranges? Such experiments would help clarify the pronounced differences between synthetic and real-world AdvOCR performance, as well as the relatively limited improvement on SmuggleBench.
  3. Necessity of OTSD and the two-stage framework: How much of the improvement is attributable to privileged visual observations and on-policy distribution matching, rather than ground-truth OCR supervision and additional training? Could the authors compare complete training pipelines, such as ground-truth SFT followed by the same GRPO refinement, or privileged-view supervised distillation followed by GRPO, using comparable data and computational budgets? This would help establish the distinct contribution of the proposed training strategy.
1 comment

Thank you for the detailed rebuttal and substantial additional experiments. I appreciate that the authors have directly addressed several of my main concerns, particularly through the joint recognition–localization evaluation, matched clean–adversarial controls, held-out generation settings, and end-to-end training comparisons. These additions significantly strengthen the empirical evidence supporting the paper, e.g.

  • Benchmark validation: The new [email protected] metric and error decomposition demonstrate that AdvSpot can distinguish recognition failures from localization failures. The paired bootstrap analysis further supports the reliability of the overall improvements ([email protected]: 24.7% to 46.0%, with a 95% CI of [14.8, 25.3] percentage points for the improvement).
  • Generalization: The experiments on 400 images generated using held-out AIGC pipelines, dot/line encoding configurations, and overlay patterns directly address my earlier concern about generalization beyond the training synthesis settings. The improvement from 7.5% to 59.3% accuracy is substantial evidence that ArmorOCR is not merely adapting to the exact configurations encountered during training.
  • Methodological attribution: The new end-to-end comparisons under identical Stage 2 GRPO refinement are particularly helpful. OTSD achieves 57.1% accuracy, compared with 54.0% for standard distillation with transformed views and 47.0% for SFT. Together with the existing visual-transfer ablation, these results provide meaningful evidence for the contributions of privileged visual observations and the proposed distillation objective.

A few questions remain relevant to assessing the paper's overall contribution:

  1. The incremental scientific value of AdvSpot. The new joint metric and failure decomposition establish useful diagnostic capabilities beyond answer accuracy. However, cross-benchmark rank reversals alone do not demonstrate that AdvSpot captures fundamentally new perceptual failure mechanisms or provides substantially broader coverage than existing adversarial OCR benchmarks. The taxonomy remains relatively small at the subtype level, and the bootstrap analysis establishes aggregate reliability rather than the stability of individual subtype conclusions. I now find the grounded evaluation contribution more convincing, but its incremental significance should still be distinguished from the addition of annotations and diagnostic metrics to an existing evaluation problem.

  2. The relationship between localization and recognition. The matched clean–adversarial experiments are informative, particularly because they reveal substantially different failure patterns. For example, mirrored-text recognition accuracy drops from 99% to 2%, while localization IoU remains almost unchanged (92.51% to 92.12%). This suggests that localization and recognition can fail largely independently. Although the joint metric is valuable for diagnosis and the localization reward contributes to training performance, the experiments do not yet establish that improved localization is a principal mechanism behind the recognition gains. Further analysis separating direct recognition improvements from benefits attributable to grounding supervision would help clarify this relationship.

  3. Methodological novelty and the scope of generalization. The additional ablations now provide convincing evidence that the proposed components improve performance, but the underlying techniques remain closely related to existing privileged-observation distillation and task-conditioned reinforcement learning. The 3.1-point improvement over standard transformed-view distillation after identical GRPO refinement is meaningful, although the extent of the distinct methodological contribution remains a consideration. Similarly, the new held-out tests establish strong generalization to unseen configurations within four known synthesis families, rather than to entirely new adversarial mechanisms. This does not invalidate the results, but the scope of the robustness claims should reflect this distinction.

Despite the remaining issues, I would like to raise my ratings to weak accept based on the improved soundness and the contribution of putting forward a useful benchmark.

Reviewer_RCLWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies adversarial visual text that remains readable to humans but challenges large multimodal models. It introduces AdvSpot, a benchmark containing 383 images and 389 grounded question–answer pairs, with bounding boxes, transcriptions, and annotations covering five categories and thirteen subtypes. It also proposes ArmorOCR, a two-stage training framework built on Qwen3-VL-8B-Instruct. First, observation-transferred self-distillation trains an original-image student using a teacher conditioned on selected transformed views and privileged textual information. Second, task-conditioned GRPO optimizes localization, recognition, full spotting, and grounded VQA.

On AdvSpot, ArmorOCR improves answer accuracy from 29.8% to 57.1% and average localization IoU from 54.4% to 63.2%. Evaluation on two additional adversarial OCR benchmarks, three general OCR benchmarks, and component ablations provides further validation.

Critical strengths
  1. The benchmark provides a useful extension beyond recognition-only evaluation. Region annotations and questions referring to specific text instances allow recognition and localization failures to be examined separately. The independent annotations, consensus filtering, and two rounds of expert review strengthen the benchmark’s credibility. This is a consequential resource contribution because answer accuracy alone can obscure failures to identify the relevant visual evidence.

  2. The method achieves large improvements with meaningful supporting ablations. The 27.3 percentage-point accuracy improvement on AdvSpot is substantial. In Table 5, removing transformed visual supervision reduces Stage 1 accuracy from 46.8% to 34.7%, even while retaining privileged textual information, supporting the contribution of visual transfer. Results on external adversarial benchmarks and the relatively small declines on general OCR benchmarks further support practical value.

Critical weaknesses
  1. The incremental methodological contribution is not sufficiently established for the final system. As acknowledged in Section 2.3, privileged-view on-policy distillation builds on existing visual self-distillation approaches. Table 5 provides an important close control: retaining transformed observations with the standard distillation objective achieves 44.2% accuracy and 54.5% IoU, versus 46.8% and 58.7% for full OTSD. Thus, much of the Stage 1 accuracy improvement is already obtained without the proposed response-region-aware objective. The paper does not show whether these differences persist after identical Stage 2 refinement, or how SFT followed by the same refinement performs. This leaves uncertainty about the final contribution of the additional methodological components.

  2. The evaluation incompletely validates the benchmark’s central grounded-perception claim. Answer accuracy and localization IoU are reported separately, so the aggregate results do not establish how often the model simultaneously transcribes the correct text and localizes its supporting region. Substring-based answer scoring also accepts additional content. These choices do not invalidate the reported improvements, but they limit how confidently those improvements can be interpreted as joint grounded correctness. This matters because grounded evaluation is the benchmark’s principal distinction from recognition-only resources.

Questions for authors
  1. Assessment-critical comparison: How do SFT and the transformed-view standard-distillation variant perform after the identical Stage 2 GRPO procedure? Does the final advantage of OTSD persist under matched refinement?

  2. Evaluation clarification: Can you report the fraction of examples with both a correct transcription and IoU ≥ 0.5, together with normalized exact-match transcription results? Do the principal model comparisons remain consistent?

  3. Reliability clarification: How stable are the differences between the standard-distillation variant and full OTSD across training runs, particularly the 2.6 percentage-point accuracy improvement? Paired per-example analysis or repeated-run results would help assess the contribution’s reliability.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.