acceptodds
Under review as a conference paper at ICLR 2027

VeriPO: Closing the Reasoning-Supervision Gap in Document Reinforcement Learning

Abstract

Reinforcement learning with rule-based, verifiable rewards has become the dominant post-training recipe for document vision-language models, yet it treats the decoded trajectory as an undifferentiated scalar-reward target. We observe that this practice clashes with the internal structure of document-QA outputs: a trajectory decomposes into parse, evidence, reason, and answer segments of sharply different verifiability, and the reasoning segment admits no rubric at all. Rule-only objectives therefore leave reasoning tokens with no within-trajectory discriminative signal, producing slow, variance-dominated training, a phenomenon we call the reasoning-supervision gap. We close this gap with Verifiability-aware Policy Optimization (VeriPO), an on-policy reinforcement-learning algorithm whose token-level advantage combines per-segment verifiable rewards with a token-level reverse-KL signal from a frozen vision-language teacher. A rubric-gated trust coefficient, computed by scoring the teacher against the same rubric the student faces, attenuates teacher supervision wherever the teacher itself fails verification. The design admits a provable verification-safe guarantee: under an adversarially uninformative teacher, VeriPO's gradient collapses onto the rule-only baseline and never underperforms it. VeriPO further trains with a single on-policy rollout per update, eliminating the group-rollout overhead of GRPO-style training. Applied to a unified document-QA decoder that emits parse, evidence, rationale, and answer in a single pass with self-citing Document Units, VeriPO advances the state of the art on multi-page document QA, chart and infographic QA, and document parsing, at a fraction of the post-training compute of group-rollout baselines. Three diagnostic studies confirm each of its mechanisms: a component ablation, an adversarial-teacher study, and a per-segment analysis that isolates reasoning as the primary beneficiary of teacher supervision.

Primary area:
applications to neuroscience & cognitive science
Keywords:
LLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_pCSDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Verifiability-aware Policy Optimization (VeriPO), a reinforcement-learning algorithm which assigns token-level credit by combining per-segment verifiable rewards with a token-level reverse-KL signal from a frozen vision-language teacher.

Critical strengths
  • The method tackles an important problem of credit assignment for reasoning tokens in RLVR for visual document QA.

  • Results show strong improvements across document QA and parsing benchmarks. The ablations support the benefit of combining rule rewards and teacher supervision, and examine the effects of routing and teacher quality.

Critical weaknesses
  • The benefit of credit assignment is not clearly seen from the ablation analysis. The rule-only ablation removes the teacher signal while retaining zero rule-reward weight on reasoning tokens. This is different from standard RLVR, where the final-answer reward also supervises reasoning tokens. A matched baseline that passes the outcome reward advantage to reasoning tokens is needed to show the benefit of VeriPO's credit assignment.

  • The efficiency comparison also needs to outline the 72B teacher cost. A compute-matched distillation baseline would better distinguish the benefit of VeriPO from distillation.

Questions for authors

Please refer to the Weaknesses section above.

Reviewer_4XdLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Please see above

Critical strengths

Please see above

Critical weaknesses

Please see above

Questions for authors

Please see above

Reviewer_AvpAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VeriPO addresses the lack of direct supervision for the reasoning process in document question answering. Given a question and retrieved page images, the 7B student model sequentially outputs a structured page parse, evidence citations, a rationale, and the final answer in a single response. The authors combine segment-level rule rewards with token-level feedback from a frozen 72B teacher model and adjust the weight of the teacher signal in the reasoning segment based on the teacher’s own answer score on the same question. During reinforcement learning, only one on-policy rollout is sampled from the student per question. The full model achieves an average accuracy of 76.04% across five document QA benchmarks, compared to 71.62% for the rule-only variant. On the Chinese and English subsets of OmniDocBench, the full model also achieves the lowest overall edit distance among the models listed in Table 1.

Critical strengths
  1. The rationale for per-segment supervision is clear. For example, while the program can check whether a number cited by the student model appears in its own parse of the table, it is difficult for the program to directly determine whether that number can be used to derive the answer. The authors supplement the hard-to-verify reasoning steps with token-level teacher feedback, while continuing to score the final answer according to the task’s own rules. This approach leverages the token-level teacher feedback on the reasoning segment while preserving clear scoring criteria.

  2. The per-segment ablation shows where the teacher signal is most helpful. Starting from the same rule-only baseline, adding the teacher signal to the parse segment alone, the evidence segment alone, or the reasoning segment alone improved average accuracy across the five benchmarks by 1.1, 1.3, and 2.7 percentage points, respectively. This ablation isolates where teacher supervision is applied, providing more direct evidence of the method’s contribution than the comparison between full VeriPO and its Qwen2.5-VL-7B backbone in Table 2.

Critical weaknesses
  1. It remains unclear which reasoning errors the student model made less often as its score improved. The evaluation counts both correct answers under sufficient context and correct abstentions under insufficient context. Aggregate accuracy alone therefore cannot distinguish between fewer calculation errors and better decisions about when to abstain. Figure 3 could have illustrated the intermediate calculations, but the last year on the horizontal axis is 2015, while the answer on the right is 2022. The figure also omits the per-year differences between the two curves described in the main text. Thus, while the results support adding the teacher signal to the reasoning segment to improve final QA performance, the evidence that the reasoning steps themselves became more reliable remains limited.

  2. The abstract’s claim regarding weak teachers goes beyond the scope of Proposition 1. Under assumptions that include the teacher’s rubric scores remaining below their respective thresholds by a fixed margin, the proposition states that as the gate sharpness tends to infinity, the teacher weights approach zero and the gradient of the VeriPO objective converges to that of the rule-only objective. In the experiment with fixed gate sharpness, the student trained with the adversarial teacher achieved 71.65% average accuracy, compared with 71.62% for the rule-only baseline. These results were close in this experiment, but they do not guarantee that final accuracy will remain at or above the rule-only baseline with other unreliable teachers.

Questions for authors
  1. In Table 2, do all baselines see the same three retrieved pages as VeriPO for each question? Can baselines with active retrieval access additional pages? Do all models receive equivalent instructions to abstain when the retrieved context is insufficient, and how are alternative phrasings of "insufficient to answer" scored?

  2. Could you compare the rule-only baseline with the variant that applies the teacher signal only to the reasoning segment on the same set of questions, examining the evidence each model selects and the intermediate calculations it performs? Please also clarify how the output labeled "UniDocVLM" in Figure 3 relates to VeriPO.

  3. In Section 4.4, when replacing the original teacher with a weaker one, are the gate thresholds retained from the original teacher or recalculated from the new teacher’s training-set rubric scores? This choice affects whether the weak-teacher experiment satisfies the teacher-score assumption in Proposition 1.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.