- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper addresses the challenge of local credit assignment during the reinforcement learning (RL) post-training phase for full-page document parsing. To overcome the limitations of conventional group-relative methods—which only yield coarse-grained, scalar, page-level advantage values—we propose the Page-to-Token Policy Optimization (P2TPO) framework. This framework integrates three core modules: (1) Reference-Element Alignment (P2BPO), which evaluates corresponding text blocks across different rollout stages; (2) Unified Anchor Pricing (UAP), which imposes penalties for content omissions and imperfect consensus on bound elements; and (3) Budget-Preserving Span Credit Redistribution (SCR), which concentrates credit for specific text blocks onto the precise spans—whether erroneous or correct—that align with the edits. Experiments conducted across 6 benchmark datasets, 13 data splits, and 5 random seeds demonstrate that P2TPO consistently outperforms response-level RL baselines, yielding robust empirical gains. Furthermore, comprehensive ablation studies are provided for each of the framework's modules.
Critical strengths
- This paper effectively identifies a critical bottleneck in full-page document parsing: page-level scalar rewards tend to dilute the impact of errors or omissions specific to individual elements. The proposed hierarchical routing method—which maps page-level utility to reference blocks and edit-level token spans—provides a sound and insightful inductive bias for structured transcription.
- The experimental validation is exceptionally comprehensive, encompassing six distinct benchmarks (across 13 splits), multiple model scales (2B/4B), various SFT initialization strategies, and training durations extended to 500 update steps. Statistical stability is ensured by averaging results across five independent reinforcement learning training seeds.
- Beyond standard component addition and removal, the approach incorporates controlled spatial credit perturbations (cyclic shifts and random support). This compellingly demonstrates the genuine advantages of token placement guided by alignment information, as opposed to relying on arbitrary, non-uniform credit weighting.
Critical weaknesses
- Although P2TPO outperforms response-level algorithms (such as GRPO, GSPO, GDPO, and RDPO)—claiming an advantage in structured credit assignment—it has not been directly compared against existing fine-grained reinforcement learning methods (e.g., CSPO or process-level reward models adapted for document elements). While current evidence suggests it is superior to coarse-grained aggregation, it has yet to demonstrate an advantage over existing paradigms for token- or sub-goal-level credit assignment.
- Ablation studies indicate that SCR provides negligible independent performance gains for P2BPO (+0.08 on OmniDocBench). In the absence of UAP (Unified Allocation Policy), it proves beneficial in only 2 out of 13 scenarios; however, when combined with UAP, it yields improvements in 11 out of 13 scenarios. The theoretical and empirical rationale for why the benefits of SCR are almost entirely contingent upon UAP remains insufficiently explored.
- Section 5 demonstrates that aggregation methods result in the loss of block-level information, whereas SCR preserves block budgets. Nevertheless, these findings are a direct consequence of the definitions involving matrix projection and normalized reweighting. The analysis lacks formal guarantees regarding the reduction of gradient variance, bounds on policy improvement, or robustness against gradient bias introduced by "ceiling" penalties.
Questions for authors
- Could the authors provide an empirical comparison against tuned fine-grained or process-level reinforcement learning baselines (e.g., CSPO adapted for multiple element types or token-level reward routing) to demonstrate that P2TPO outperforms existing structured credit assignment mechanisms, rather than merely surpassing response-level aggregation?
- Can the authors clarify the underlying optimization mechanism? Specifically, does the UAP facilitate effective gradient routing for SCR by altering the advantage distribution—for instance, through changes in sparsity, variance, or sign balancing (achieved via omission handling and consensus penalties)?
- Section 5 establishes both the information loss resulting from aggregation (Proposition D.1) and the preservation of credit (Proposition D.4). However, the preservation of credit does not theoretically guarantee policy improvement or a reduction in variance. Does P2TPO formally optimize a well-defined surrogate policy objective? How does the upper-bound gap penalty within the UAP affect the bias and variance of the policy gradient estimates? Furthermore, could the authors discuss whether the concentration of credit in SCR poses a risk of gradient instability during the early stages of reinforcement learning training?
3 comments
Thanks for your comments and the extra experiment. The result is good, I'll keep my score as weak acceptance.
Thank you for taking the time to review our response and additional experiment. We truly appreciate your positive feedback and the thoughtful attention you gave to our work.
Response to Reviewer 4ktu
Thank you for your detailed review and constructive questions. We address your concerns below.
Fine-grained comparisons (W1/Q1)
We test type-level routing and token-level supervision. P2BPO-Type shares a coefficient across blocks of the same type to replace the CSPO; P2BPO compares each reference-block instance across rollouts. This retains local quality differences that type pooling can cancel. OPSD provides token supervision from a frozen, reference-conditioned teacher on student-generated prefixes, giving a process-level baseline.
Tables 1/11; five-run means±SD. Routing variants share initialization/budget, use local-only feedback, and disable UAP/SCR.
UAP–SCR interaction (W2/Q2)
UAP changes both the availability and sign of credit. SCR emphasizes errors for negative coefficients and correct positions for positive coefficients. UAP includes matched scores and zero-valued omissions in each reference block’s group. With mean μ and population standard deviation σ, it assigns (Qi−μ)/σ when σ>ϵ, and −λ(1−μ) otherwise.
In the relative branch, adding omissions can lower the mean and turn matched-block credit from negative to positive, switching SCR’s target. At identical imperfect scores, UAP supplies negative credit where matched-only normalization gives zero. SCR can then redistribute it within eligible blocks. Whole-block omissions use separate boundary proxies.
OmniDocBench Overall, five-run means±SD; Tables 1/3.
Fixed-response replay supports this mechanism: 108/447 (24.2%) initially nonzero matched-only coefficients switch from negative to positive in nondegenerate groups containing zero-valued outcomes, used as the omission proxy. The median completed-to-matched-only absolute-coefficient ratio is 0.888 (App. C.3). UAP changes routing rather than simply amplifying credit; this supports complementarity without isolating all sources of the final gains.
Objective, guarantees, and stability (W3/Q3)
P2TPO optimizes the explicit surrogate in Eq. (4):
J(θ)=∑iTi1∑i,tℓdual(ρi,t(θ),Ai,t).
Here, Ti is the response length, ρ is the token probability ratio, and ℓdual is the dual-clipped PPO surrogate. Composed coefficients and old-policy probabilities stay fixed during updates. At the sampling policy, the gradient averages Ai,t∇logπθ over tokens. Redistribution can change its direction at a fixed block budget because token gradients differ.
For eligible matched blocks with nonzero credit and nonempty, nonfull masks, Proposition D.4 gives a fixed target’s absolute-credit share as q+η(ϱ−q), where q is target prevalence, ϱ mask precision, and η effective redistribution strength. It exceeds uniform allocation exactly when η>0 and ϱ>q, before page/omission composition. Circular-shift and random-support controls test placement at a fixed weight multiset and budget (Table 3).
Controlled ablations test calibration and credit conservation:
Table 3; five-run means±SD.
The NGRPO-style control replaces only low-dispersion branches with virtual-maximum normalization (App. C.3.1). The conservation control reduces cap-active block budgets at fixed relative token weights. These results support both quality-gap calibration and the preservation of block credit during redistribution.
The ceiling penalty is bounded by λ and grows with the quality gap. This outcome-dependent shaping need not preserve the expected page-return gradient. Variance also depends on coefficient second moments and token-gradient covariance.
In early training, few or inaccurate masks can concentrate credit on unhelpful positions. We use a 0.8 quality gate to avoid this (App. D.3; Tables 6/8).
We directly check this risk using pre-clip gradient norms:
Recorded gradients remain below the clipping threshold in 5 seeds, including early training. We will add this analysis in the PDF if time allows.
We hope these explanations and results address your concerns. Thank you again for your helpful feedback.