acceptodds
Withdrawn from ICLR 2027

Improving OCR Faithfulness via Gated and Attenuated On-Policy Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language models may rewrite anomalous text in images into linguistically plausible expressions, compromising OCR transcription faithfulness. Sequence-level task rewards and local teacher guidance are complementary, but guidance from the same teacher may not remain equally effective as the student improves. Offline analysis shows that supervision from a fixed teacher becomes progressively less favorable as the student improves, both across training checkpoints and across response groups with different task rewards. Motivated by this observation, we introduce GAD-RL, which adaptively regulates teacher supervision during joint post-training according to the student's current task performance and local distributions. A frozen teacher conditions on reference transcriptions and student-generated prefixes. GAD-RL disables distillation for response groups containing an output with task reward at least 0.95 and continuously attenuates distillation strength as group-mean reward increases. It also weights forward KL by the student's probability of the teacher's Top-1 token, moderating local auxiliary updates when student support for that candidate is low. On Qwen3.5-2B, GAD-RL achieves 59.92% Micro Recall on CHAOS-Bench, surpassing GRPO and GRPO+OPD (fixed-weight) by 8.45 and 4.43 percentage points, respectively, while achieving an Overall score of 91.18 on OmniDocBench v1.6.

Primary area:
reinforcement learning
Keywords:
OCR Faithfulness, Document Parsing, Vision-Language Models, On-Policy Distillation, Reinforcement Learning, Linguistic-Prior Hallucination

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DYkMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposed GAD-RL to address a problem where VLM models always unconsciously correct the original wrong words to plausible right words

Critical strengths
  1. idea is straight-forward and the motivation is reasonable
  2. experiment design is good
Critical weaknesses
  1. If teacher model needs to take a glance of GT, why not just using GT to build token guidance (no teacher model is needed)
  2. no code; I highly recommend authors provide a small demo that at least can match some numbers described in the paper; I'll run it
  3. The “Teacher errors” values in Table 5 are exactly the same in the “Incompatible” column of Table 9; I think authors need to explain this
  4. Section 5.2said the fixed-weight OPD also uses SA-FKL; it gets a score of 0.5549. In Table 4, the “w/o RAA” entry (i.e., gating + SA-FKL only) achieved a score of 0.5414, which is lower than that of the fixed-weight method; Can authors explain this?
  5. The SNR dropped from 12.57 to 0.114, mainly because erroneous tokens have almost disappeared, not because the teacher has gotten worse. This is the data on which the core motivation is based, and it has been exaggerated
Questions for authors
  1. If teacher model needs to take a glance of GT, why not just using GT to build token guidance (no teacher model is needed)
  2. no code; I highly recommend authors provide a small demo that at least can match some numbers described in the paper; I'll run it
  3. The “Teacher errors” values in Table 5 are exactly the same in the “Incompatible” column of Table 9; I think authors need to explain this
  4. Section 5.2said the fixed-weight OPD also uses SA-FKL; it gets a score of 0.5549. In Table 4, the “w/o RAA” entry (i.e., gating + SA-FKL only) achieved a score of 0.5414, which is lower than that of the fixed-weight method; Can authors explain this?
  5. The SNR dropped from 12.57 to 0.114, mainly because erroneous tokens have almost disappeared, not because the teacher has gotten worse. This is the data on which the core motivation is based, and it has been exaggerated
  6. “Harmful signals” may be a false impression caused by tokenization drift. If this explanation is true, the correct adjustment would be to align the segmentation of teachers and students, rather than reducing the number of teacher tokens
1 comment

Reply 5:

We thank the reviewer for highlighting the role of changes in error proportions. However, the teacher model remains frozen throughout training. Therefore, our motivation is not based on the assumption that the teacher’s own capability deteriorates, but rather on understanding how the effectiveness of its supervision changes as the student’s generation trajectories evolve. We agree that the overall B/H ratio depends not only on the proportions of correct and incorrect tokens in the student’s outputs, but also on the teacher’s conditional supervision behavior at correct and incorrect positions. To disentangle these factors, we conducted a two-factor reweighting analysis. Specifically, when replacing the proportions of correct and incorrect tokens in the Base model with those observed at training step 300, while keeping the within-category supervision-signal rates of the Base model unchanged, the B/H ratio decreases from 12.57 to 1.656, rather than the actually observed value of 0.114. This reweighting shows that the changing error proportion alone does not reproduce the observed B/H ratio; the correctness-conditioned directional signal rates also change as the student evolves.

Reply 6:

We thank the reviewer for raising this concern. Both the student and teacher models used in this analysis are based on Qwen3.5-2B. We have verified that all response token IDs involved in the evaluation decode to exactly the same content under both models’ tokenizers. During teacher scoring, we directly concatenate the original response token IDs generated by the student to the tokenized privileged prompt, without first decoding the response into text and then re-tokenizing it. Therefore, both models score the same actual generated token IDs under identical response token prefixes. The changes in token counts discussed in Section 5.1 refer to differences in the outputs generated by the student across training checkpoints, rather than inconsistencies in tokenization between the teacher and student when scoring the same response. GAD-RL does not modify the tokenization scheme or reduce the number of tokens required to represent a given response. Instead, it achieves adaptive distillation control by regulating whether teacher supervision is applied at student-generated token positions and how strongly that supervision is applied. Our compatibility analysis in Table 9 explicitly allows alternative tokenizations: a teacher Top-1 candidate is classified as GT-compatible if its decoded content matches a prefix of the GT continuation. Thus, a different tokenization alone is not counted as a teacher content error.

Reviewer_bL6WOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes GAD-RL, a post-training method for improving OCR faithfulness when visible text conflicts with linguistic priors. It combines GRPO with on-policy distillation from a frozen, same-backbone teacher fine-tuned on 200,000 text transcription examples. The teacher receives the reference transcription and scores student-generated prefixes. Distillation is disabled for groups containing a sufficiently high-reward response, attenuated continuously by group-average reward, and locally weighted by the student's probability of the teacher's top-ranked token. Evaluation on two 2B backbones covers CHAOS-Bench, GlitchText, and OmniDocBench, with component ablations and offline analyses of teacher supervision.

Critical strengths
  1. The paper addresses a meaningful OCR failure: replacing anomalous printed text with linguistically plausible alternatives. Table 1 reports CHAOS-Bench recall improvements over GRPO of 8.45 and 3.82 percentage points on Qwen3.5-2B and Qwen3-VL-2B, respectively, while maintaining comparable general document-parsing performance. The results support practical value within the studied setting, particularly on Qwen3-VL-2B, where fixed-weight OPD underperforms GRPO but GAD-RL exceeds GRPO.
  2. The study includes substantial supporting controls. It examines coefficients and distillation objectives, removes individual components, compares linear decay and low-score selection, and varies the teacher. Removing reward-aware attenuation reduces recall from 59.92% to 54.14%, and removing student-probability weighting reduces it to 55.98%. Table 7 supports the usefulness of the transcription-trained teacher within this framework. The teacher top-1 analysis and within-checkpoint reward stratification also strengthen the empirical motivation beyond a simple probability-disagreement statistic.
Critical weaknesses
  1. Limited methodological novelty beyond existing adaptive on-policy distillation. GKD already supports OPD with RL; OPSD uses privileged reference solutions at student-generated prefixes; KDRL studies GRPO with distillation, annealing, and any-success group masking; and I-SDPO adjusts reliance on privileged distillation according to group success. The manuscript acknowledges these antecedents. Its methodological contribution therefore rests mainly on the particular continuous reward attenuation and local probability weighting, together with the OCR-specific evidence. The existing ablations provide relevant support: removing RAA is already close to adapting KDRL's group-masking variant to the same teacher and local loss. Nevertheless, I find the demonstrated methodological increment modest, making convincing evidence for the distinctive benefit of the proposed weighting scheme particularly important.

  2. Incomplete attribution and robustness evidence for the remaining gains. The advantage over the stronger simple controls is 2.99 percentage points over linear decay and 2.17 points over low-score OPD, with no reported variability across training runs for these differences. Furthermore, the experiments do not fully separate the benefit of allocating distillation weights according to group reward from the benefit of weakening distillation overall. The mean attenuation factor among active groups falls from approximately 0.12 to 0.02, while gating and token weights further affect the applied strength. The fixed-coefficient sweep and the 58.82% result with a tenfold GAD-RL coefficient provide useful partial evidence, but do not match the effective strength profile of a reward-agnostic control. A strength-matched comparison and repeated-run results for the closest baselines would help establish whether the proposed state-dependent allocation provides a reliable additional benefit.\n\nThe reported gains make the approach promising, but the limited demonstrated methodological increment and unresolved attribution lead me to recommend weak rejection.

Questions for authors
  1. Does reward-dependent attenuation retain its advantage over a reward-agnostic control with a comparable effective distillation-strength profile, using the same teacher and local loss? For example, permuting the attenuation weights among active groups within a training step would preserve their distribution while testing whether their association with each group's reward matters.
  2. How stable are the main gains, especially 59.92% versus 56.93% and 57.75%, across independent training runs? Please report the number of runs and suitable uncertainty for the closest comparisons. Repeating every ablation is not necessary.
  3. As a secondary question about practical value, how does the method compare with GRPO plus direct reference-text cross-entropy supervision, and what is the training cost including teacher construction and scoring? This would test a simpler auxiliary-supervision alternative; it would not by itself isolate the benefit of student-generated prefixes. Table 7 already supports teacher usefulness within GAD-RL, so my question concerns marginal benefit and cost, not whether a teacher is logically necessary.
1 comment

Reply Q3:

We agree that the key question is the marginal benefit and computational cost relative to simpler auxiliary supervision. Comparison with GRPO+SFT. We additionally evaluated GRPO+SFT, which jointly optimizes the GRPO loss and a standard supervised fine-tuning loss for document parsing. The SFT loss uses the same coefficient as the distillation loss, with all other training hyperparameters unchanged. On Qwen3.5-2B, the CHAOS-Bench Micro Recall results are:

MethodMicro Recall
GRPO51.47%
GRPO+SFT51.96%
GAD-RL59.92%
GAD-RL therefore outperforms this direct-supervision baseline by 7.96 percentage points, while GRPO+SFT improves over GRPO by 0.49 points. This comparison supports the practical benefit of our auxiliary-supervision approach under the evaluated settings.
Training cost, including teacher scoring. GRPO takes approximately 220 seconds per training step, while GAD-RL takes approximately 260 seconds, including teacher scoring. Thus, the measured training-time overhead is approximately 18.2%. Over 300 steps, these timings correspond to approximately 18.3 hours for GRPO and 21.7 hours for GAD-RL, excluding teacher construction. The teacher remains frozen and performs forward scoring only, requiring no teacher-side backpropagation or optimizer states.
One-time teacher construction and reuse. Training the 2B teacher takes approximately 10 hours, or 80 GPU-hours. This cost is separate from the per-step overhead above and can be amortized across subsequent student-training runs with compatible tokenizers and vocabularies. For example, we also use Qwen3.5-2B as the teacher for Qwen3.5-9B, achieving 63.53% Micro Recall on CHAOS-Bench (see our response to Reviewer 9su7, Q3). This demonstrates that the small teacher can be reused when training a larger student.
Together, these results quantify the accuracy gain over the tested GRPO+SFT alternative and the additional cost relative to GRPO: 7.96 percentage points higher Recall, approximately 18.2% additional student-training time, and a one-time teacher-construction cost of 80 GPU-hours.
Reviewer_9su7OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GAD-RL, a joint GRPO + on-policy distillation framework for OCR transcription faithfulness. A frozen teacher supplies token-level supervision. The authors propose 1) Mastery-Gated Distillation (MGD) disables distillation on groups where any sampled response already hits task reward ≥ 0.95; 2) Reward-Aware Attenuation (RAA) continuously decays distillation strength as group-mean reward rises; and 3) Student-Adaptive Forward KL (SA-FKL) weights the forward-KL loss by the student probability at the teacher's Top-1 token.

Critical strengths
  1. The core observation is empirically interesting and well visualized in Figure 1 and Figure 4.
  2. The ablation table separately removes SW, MGD, and RAA, which is a reasonable attempt to disentangle the three contributions.
Critical weaknesses
  1. The frozen teacher is conditioned on the ground-truth transcription z. It is unclear what the teacher adds over using z directly to build a token-level target. A GT-only baseline (no teacher model) is missing and would be the natural control for whether the "teacher" even needs to exist.
  2. Only two small backbones are evaluated (Qwen3.5-2B and Qwen3-VL-2B). The gain on Qwen3-VL-2B is less than half the gain on Qwen3.5-2B, and the OmniDocBench Overall actually drops from 89.86 (GRPO) to 90.00 (GAD-RL) while fixed-weight GRPO+OPD drops to 87.67. The claim of "comparable general parsing performance" obscures that improvements are task-specific and backbone-sensitive.
  3. The MGD component contributes very little: Table 4 shows "w/o MGD" at 0.5931 vs. full GAD-RL at 0.5992, i.e., only 0.61 pp. The paper's three-component story is really a one-component story (RAA), with SW as a secondary knob and MGD mostly cosmetic.
Questions for authors
  1. Since the teacher is conditioned on the GT transcription (without the image), what does the teacher distribution give you that cannot be recovered by directly using z and the student's own prefix to construct token-level targets? A no-teacher GT-only ablation would clarify this.
  2. The gap between Qwen3.5-2B and Qwen3-VL-2B is large. Is this a backbone-capacity effect, a tokenizer/vocabulary effect, or an artifact of different teacher-student KL scale?
  3. Would GAD-RL still help at 7B+ scale where teacher-student gaps are smaller?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Responses and Revisions

We thank the reviewers for their constructive feedback. We summarize the additional evidence and manuscript revisions below.

  1. Clarifying the motivation. Our analysis examines how a frozen teacher’s supervision changes relative to the student’s evolving outputs. Replacing the Base correctness proportions with those at step 300, while retaining the Base conditional signal rates, changes B/H from 12.57 to 1.656, compared with the observed 0.114. This identifies two contributors to the decline: fewer opportunities to correct student errors and changes in correctness-conditioned supervision. These findings motivate adjusting teacher intervention as student performance improves.
  2. Attributing the gains to reward-dependent allocation. We shuffled group-mean rewards among active distillation groups solely for computing RAA, keeping the teacher, SA-FKL, gate, and base coefficient fixed. GRPO and gating retained the original rewards. This preserves the exact multiset of attenuation weights within each batch while changing which group receives each weight. Recall falls from 59.92% to 53.73%, a 6.19-point decrease. This controlled comparison supports attributing the benefit to matching supervision strength to each group’s performance: stronger guidance for lower-reward active groups and weaker guidance for higher-reward groups. Our point-by-point response also reports two independent training runs per configuration; GAD-RL outperforms Linear-decay OPD and Low-score OPD in both runs.
  3. Evaluating direct GT supervision. The reference-conditioned teacher converts GT and the student’s generated prefix into next-token supervision. We added edit-aligned GT-only and GRPO+SFT controls, obtaining 42.56% and 51.96% Recall, respectively, versus 59.92% for GAD-RL. These comparisons support the practical value of teacher-mediated supervision under the evaluated settings.
  4. Extending evaluation to a 9B student. Qwen3.5-9B uses the transferred hyperparameters and the same Qwen3.5-2B SFT teacher. GAD-RL improves CHAOS-Bench Micro Recall from 53.82% to 63.53% (+9.71 pp) and OmniDocBench Overall from 92.70 to 93.44. These results extend the evidence for improved transcription faithfulness with preserved general parsing performance, with gain magnitudes varying across backbones and tasks.
  5. Clarifying component roles and prior work. Our central contribution is the OCR supervision-dynamics analysis and continuous reward-dependent regulation of teacher intervention. MGD follows the success-based group-masking idea explored in prior work such as KDRL. It provides supplementary gains of 0.61 and 0.98 pp at coefficients 0.005 and 0.05, respectively. Student-probability weighting moderates local FKL updates; removing it reduces Recall by 3.94 and 5.68 pp. The revised discussion distinguishes these roles and their measured contributions.

Training costs, the anonymous demo, tokenization consistency, and diagnostic definitions are detailed in the corresponding point-by-point responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.