acceptodds
Under review as a conference paper at ICLR 2027

P2TPO: Structure-Aligned Page-to-Token Credit Assignment for Full-Page Document Parsing

Abstract

End-to-end full-page document parsing must preserve the content and structure of heterogeneous elements, yet otherwise accurate outputs can contain local errors and omissions. A single page-level advantage does not explicitly identify which elements differ in quality across sampled outputs or where their errors occur. We introduce Page-to-Token Policy Optimization (P2TPO), a structure-aligned credit-assignment framework that operates on full-page policy trajectories. Reference-element alignment first associates block-level quality contrasts with generated spans. Unified Anchor Pricing (UAP) incorporates matched and missing element outcomes and provides feedback relative to a fixed quality ceiling when sampled outputs receive identical but imperfect scores for a reference element. Within matched blocks, Span Credit Redistribution (SCR) redistributes existing block credit according to aligned correct and erroneous spans, using boundary proxies for within-block omissions while preserving the total block-derived advantage across its tokens. Under a fixed-weight additive page utility, we show that element-indexed contrasts retain local quality differences that page-level aggregation can cancel out. In the main experimental comparison, P2TPO outperforms all evaluated policy optimization baselines in seed-averaged scores on each of the six document parsing benchmarks. Additional experiments demonstrate gains across model scales and initializations.

Primary area:
reinforcement learning
Keywords:
VLM, document parsing, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4ktuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper addresses the challenge of local credit assignment during the reinforcement learning (RL) post-training phase for full-page document parsing. To overcome the limitations of conventional group-relative methods—which only yield coarse-grained, scalar, page-level advantage values—we propose the Page-to-Token Policy Optimization (P2TPO) framework. This framework integrates three core modules: (1) Reference-Element Alignment (P2BPO), which evaluates corresponding text blocks across different rollout stages; (2) Unified Anchor Pricing (UAP), which imposes penalties for content omissions and imperfect consensus on bound elements; and (3) Budget-Preserving Span Credit Redistribution (SCR), which concentrates credit for specific text blocks onto the precise spans—whether erroneous or correct—that align with the edits. Experiments conducted across 6 benchmark datasets, 13 data splits, and 5 random seeds demonstrate that P2TPO consistently outperforms response-level RL baselines, yielding robust empirical gains. Furthermore, comprehensive ablation studies are provided for each of the framework's modules.

Critical strengths
  1. This paper effectively identifies a critical bottleneck in full-page document parsing: page-level scalar rewards tend to dilute the impact of errors or omissions specific to individual elements. The proposed hierarchical routing method—which maps page-level utility to reference blocks and edit-level token spans—provides a sound and insightful inductive bias for structured transcription.
  2. The experimental validation is exceptionally comprehensive, encompassing six distinct benchmarks (across 13 splits), multiple model scales (2B/4B), various SFT initialization strategies, and training durations extended to 500 update steps. Statistical stability is ensured by averaging results across five independent reinforcement learning training seeds.
  3. Beyond standard component addition and removal, the approach incorporates controlled spatial credit perturbations (cyclic shifts and random support). This compellingly demonstrates the genuine advantages of token placement guided by alignment information, as opposed to relying on arbitrary, non-uniform credit weighting.
Critical weaknesses
  1. Although P2TPO outperforms response-level algorithms (such as GRPO, GSPO, GDPO, and RDPO)—claiming an advantage in structured credit assignment—it has not been directly compared against existing fine-grained reinforcement learning methods (e.g., CSPO or process-level reward models adapted for document elements). While current evidence suggests it is superior to coarse-grained aggregation, it has yet to demonstrate an advantage over existing paradigms for token- or sub-goal-level credit assignment.
  2. Ablation studies indicate that SCR provides negligible independent performance gains for P2BPO (+0.08 on OmniDocBench). In the absence of UAP (Unified Allocation Policy), it proves beneficial in only 2 out of 13 scenarios; however, when combined with UAP, it yields improvements in 11 out of 13 scenarios. The theoretical and empirical rationale for why the benefits of SCR are almost entirely contingent upon UAP remains insufficiently explored.
  3. Section 5 demonstrates that aggregation methods result in the loss of block-level information, whereas SCR preserves block budgets. Nevertheless, these findings are a direct consequence of the definitions involving matrix projection and normalized reweighting. The analysis lacks formal guarantees regarding the reduction of gradient variance, bounds on policy improvement, or robustness against gradient bias introduced by "ceiling" penalties.
Questions for authors
  1. Could the authors provide an empirical comparison against tuned fine-grained or process-level reinforcement learning baselines (e.g., CSPO adapted for multiple element types or token-level reward routing) to demonstrate that P2TPO outperforms existing structured credit assignment mechanisms, rather than merely surpassing response-level aggregation?
  2. Can the authors clarify the underlying optimization mechanism? Specifically, does the UAP facilitate effective gradient routing for SCR by altering the advantage distribution—for instance, through changes in sparsity, variance, or sign balancing (achieved via omission handling and consensus penalties)?
  3. Section 5 establishes both the information loss resulting from aggregation (Proposition D.1) and the preservation of credit (Proposition D.4). However, the preservation of credit does not theoretically guarantee policy improvement or a reduction in variance. Does P2TPO formally optimize a well-defined surrogate policy objective? How does the upper-bound gap penalty within the UAP affect the bias and variance of the policy gradient estimates? Furthermore, could the authors discuss whether the concentration of credit in SCR poses a risk of gradient instability during the early stages of reinforcement learning training?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 4ktu

Thank you for your detailed review and constructive questions. We address your concerns below.

Fine-grained comparisons (W1/Q1)

We test type-level routing and token-level supervision. P2BPO-Type shares a coefficient across blocks of the same type to replace the CSPO; P2BPO compares each reference-block instance across rollouts. This retains local quality differences that type pooling can cancel. OPSD provides token supervision from a frozen, reference-conditioned teacher on student-generated prefixes, giving a process-level baseline.

ComparisonHigher-mean splitsResult
Block vs. type routing8/13PureDocBench Clean/Digital/Real: +0.46/+0.44/+0.26
P2TPO vs. OPSD13/13OmniDocBench: 93.12±0.23 vs. 92.31±0.16

Tables 1/11; five-run means±SD. Routing variants share initialization/budget, use local-only feedback, and disable UAP/SCR.

UAP–SCR interaction (W2/Q2)

UAP changes both the availability and sign of credit. SCR emphasizes errors for negative coefficients and correct positions for positive coefficients. UAP includes matched scores and zero-valued omissions in each reference block’s group. With mean μ\mu and population standard deviation σ\sigma, it assigns (Qi−μ)/σ(Q_i-\mu)/\sigma when σ>ϵ\sigma>\epsilon, and −λ(1−μ)-\lambda(1-\mu) otherwise.

In the relative branch, adding omissions can lower the mean and turn matched-block credit from negative to positive, switching SCR’s target. At identical imperfect scores, UAP supplies negative credit where matched-only normalization gives zero. SCR can then redistribute it within eligible blocks. Whole-block omissions use separate boundary proxies.

UAPWithout SCRWith SCRGainImproved splits
Off92.48±0.1592.56±0.21+0.082/13
On92.84±0.3093.12±0.23+0.2811/13

OmniDocBench Overall, five-run means±SD; Tables 1/3.

Fixed-response replay supports this mechanism: 108/447 (24.2%) initially nonzero matched-only coefficients switch from negative to positive in nondegenerate groups containing zero-valued outcomes, used as the omission proxy. The median completed-to-matched-only absolute-coefficient ratio is 0.888 (App. C.3). UAP changes routing rather than simply amplifying credit; this supports complementarity without isolating all sources of the final gains.

Objective, guarantees, and stability (W3/Q3)

P2TPO optimizes the explicit surrogate in Eq. (4):

J^(θ)=1∑iTi∑i,tℓdual(ρi,t(θ),Ai,t).\widehat J(\theta)=\frac{1}{\sum_iT_i}\sum_{i,t}\ell_{\mathrm{dual}}(\rho_{i,t}(\theta),A_{i,t}).

Here, TiT_i is the response length, ρ\rho is the token probability ratio, and ℓdual\ell_{\mathrm{dual}} is the dual-clipped PPO surrogate. Composed coefficients and old-policy probabilities stay fixed during updates. At the sampling policy, the gradient averages Ai,t∇log⁡πθA_{i,t}\nabla\log\pi_\theta over tokens. Redistribution can change its direction at a fixed block budget because token gradients differ.

For eligible matched blocks with nonzero credit and nonempty, nonfull masks, Proposition D.4 gives a fixed target’s absolute-credit share as q+η(ϱ−q)q+\eta(\varrho-q), where qq is target prevalence, ϱ\varrho mask precision, and η\eta effective redistribution strength. It exceeds uniform allocation exactly when η>0\eta>0 and ϱ>q\varrho>q, before page/omission composition. Circular-shift and random-support controls test placement at a fixed weight multiset and budget (Table 3).

Controlled ablations test calibration and credit conservation:

VariantOmniDocBenchSplits below full
Full P2TPO93.12±0.23—
w/ NGRPO calibration92.90±0.159/13
w/o credit conservation92.81±0.1513/13

Table 3; five-run means±SD.

The NGRPO-style control replaces only low-dispersion branches with virtual-maximum normalization (App. C.3.1). The conservation control reduces cap-active block budgets at fixed relative token weights. These results support both quality-gap calibration and the preservation of block credit during redistribution.

The ceiling penalty is bounded by λ\lambda and grows with the quality gap. This outcome-dependent shaping need not preserve the expected page-return gradient. Variance also depends on coefficient second moments and token-gradient covariance.

In early training, few or inaccurate masks can concentrate credit on unhelpful positions. We use a 0.8 quality gate to avoid this (App. D.3; Tables 6/8).

We directly check this risk using pre-clip gradient norms:

UpdatesMaximum normSeeds exceeding 1.0
Steps 1–500.53820/5
Full trajectory0.89260/5

Recorded gradients remain below the clipping threshold in 5 seeds, including early training. We will add this analysis in the PDF if time allows.

We hope these explanations and results address your concerns. Thank you again for your helpful feedback.

Thanks for your comments and the extra experiment. The result is good, I'll keep my score as weak acceptance.

Thank you for taking the time to review our response and additional experiment. We truly appreciate your positive feedback and the thoughtful attention you gave to our work.

Reviewer_HuN5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Page-to-Token Policy Optimization (P2TPO), a reinforcement-learning optimization method for full-page document parsing. Existing page-level RL methods typically assign rewards to the whole output, making it difficult to identify which specific blocks or tokens are responsible for errors. P2TPO addresses these limitations through three components: Page-to-Block Policy Optimization (P2BPO), which aligns prediction blocks with reference elements and compares corresponding blocks across rollouts; Unified Anchor Pricing (UAP), which handles cases where candidate blocks receive similar but imperfect scores; and Span Credit Redistribution (SCR), which reallocates block-level credit within matched spans according to token-level correctness while preserving the block's total credit budget. Experiments across six benchmarks and 13 evaluation splits show that P2TPO outperforms all evaluated policy optimization baselines, which demonstrates the effectiveness of the proposed method.

Critical strengths
  1. The proposed token-level credit-assignment framework outperforms conventional response-level feedback in document parsing by providing localized, structure-aware error identification. Furthermore, its core principles are generalizable to broader reinforcement learning domains.
  2. The experimental evaluation is broad, covering six document-parsing benchmarks and 13 evaluation splits. Notably, the experiments also include real-world document capture scenarios, such as pages affected by photographic distortion or geometric deformation, rather than focusing exclusively on clean, well-structured documents. These results across diverse settings underscore the method’s practical applicability.
Critical weaknesses
  1. Since P2TPO relies on ground-truth annotations to compute its training rewards, the paper does not sufficiently justify why reinforcement learning is preferable to directly continuing supervised fine-tuning on the same annotated data. The experiments demonstrate improvements over the SFT-initialized checkpoint and several RL baselines, but they do not include a continued-SFT baseline using the same training data and a comparable computational budget.
  2. The paper does not sufficiently specify the alignment algorithm used to match predicted blocks with reference elements. In particular, it remains unclear how the corresponding prediction blocks are identified for each reference element, especially in challenging cases involving duplicated content, out-of-order predictions, or ambiguous matches. Furthermore, the paper lacks sufficient detail on how token-level correctness/error masks are assigned to prediction blocks based on the reference elements.
  3. The novelty relative to existing fine-grained reward and credit-assignment methods could be established more convincingly. The related-work discussion positions P2TPO against fine-grained methods such as CSPO, but the empirical comparisons primarily involve response-level baselines. Stronger controlled comparisons with the closest fine-grained alternatives would make the novelty claim more compelling.
Questions for authors
  1. Could the authors provide more details about the alignment algorithm, including the matching criteria, ambiguity resolution, and handling of one-to-many or many-to-one correspondences?
  2. Could the authors provide controlled comparisons with the closest fine-grained alternatives under comparable experimental settings to better isolate the benefits of P2TPO's structure-aligned credit assignment and establish its methodological novelty?
  3. Given that P2TPO requires ground-truth annotations to construct its rewards, could the authors clarify the motivation for using RL instead of continued SFT? In particular, could they compare P2TPO against continued SFT starting from the same initialization, using the same annotated training data and a comparable computational budget?
5 comments
AuthorsOpenReview ↗

Response to Reviewer HuN5

We appreciate your comprehensive summary of P2TPO, particularly your recognition of the strengths in our localized, structure-aware credit assignment and extensive evaluation. We also value your constructive suggestions regarding the continued-SFT comparison, alignment methodology, and fine-grained control mechanisms.

We address W1/Q3 and W3/Q2 in the following sections. Part 2 details the alignment rules, token-mask construction, and independent audits as requested in W2/Q1.

Part 1. Why RL, and what does P2TPO add?

P2TPO increases the 13-split mean by 3.14 points compared to continued SFT, maintains a 0.94-point advantage on OmniDocBench after 500 updates, and surpasses OPSD on all 13 splits while utilizing approximately 48% less GPU time.

RL versus continued SFT (W1/Q3)

Employing identical annotations does not yield equivalent learning signals. Continued SFT (cSFT) optimizes the likelihood of reference tokens conditioned on reference prefixes. In contrast, P2TPO evaluates generated responses and assigns credit to their constituent tokens, ensuring that feedback directly corresponds to errors encountered during generation.

Both methods are compared using shared initialization, data, and update counts, with learning rates selected independently from four candidates (App. C.1.5/C.6).

StudyShared initializationAnnotated recordsPages/updateUpdatesRuns/method
PrimaryQwen3.5-2B/700K SFT12,8001281005
ExtendedQwen3.5-2B/50K SFT64,0001285005
Study / evaluationcSFTP2TPO
Primary: 13-split mean81.5184.65
Primary: OmniDocBench92.35±0.0693.12±0.23
Primary: PureDocBench Real64.16±0.2570.33±0.70
Extended: OmniDocBench, update 10090.77±0.1091.38±0.42
Extended: OmniDocBench, update 50090.55±0.1091.49±0.65

Split scores are reported as mean ± standard deviation. In the primary study, P2TPO achieves higher means on all 13 splits. The aggregate gain is 3.14, with a pointwise 95% Welch confidence interval of [2.61, 3.67] and a Holm-adjusted p<0.001p<0.001 (Table 1; App. C.1.4).

Extended training does not reduce the OmniDocBench performance gap; instead, the difference increases from 0.61 to 0.94 points. P2TPO achieves the highest mean among six methods at 45 out of 50 evaluated checkpoints (Fig. 6; Table 15; App. C.8).

Novelty and controlled comparisons (W3/Q2)

P2TPO compares identical reference blocks across rollouts, thereby preserving distinctions that pooling may obscure. Within a single response, two text blocks may simultaneously improve and deteriorate; a shared type-level coefficient cannot capture both changes. CSPO groups tokens according to table functional attributes. P2BPO-Type offers a task-adapted mechanism for controlling granularity, rather than replicating the full CSPO approach.

Two additional operations address feedback within each reference-block group. When all rollouts exhibit the same imperfect quality rr, centering results in zero relative feedback. UAP instead assigns −λ(1−r)-\lambda(1-r). SCR emphasizes error spans for negative coefficients and correct tokens for positive coefficients, while maintaining the source block’s credit budget (App. D).

The following five-run controls utilize shared initialization, data, 100 updates, and eight responses per page (Tables 3/11).

ComparisonWhat changesSplits with higher P2TPO-side mean
P2BPO vs. P2BPO-TypeBlock vs. type routing; UAP/SCR off8/13
Full vs. no omission routingUnmatched-reference routing11/13
Full vs. NGRPO calibrationLow-dispersion calibration branch9/13
Aligned SCR vs. circular shiftToken positions; same weights and block-credit budget10/13
Aligned SCR vs. random supportToken positions; same weights and block-credit budget12/13

These interventions independently support reference-block comparison, omission routing, quality-gap calibration, and correctness-aligned token placement.

The OPSD comparison also distinguishes between on-policy sampling and parsing-reward feedback. OPSD utilizes teacher–student log-probability differences. In our implementation, over 60 development experiments informed the final configuration: a frozen reference-conditioned teacher, one step per rollout batch, and a learning rate of 5×10−85\times10^{-8} (App. C.1.5/C.6).

MethodApprox. GPU-hours / 100 updates
GRPO59
P2TPO60
OPSD115

With eight responses per page, P2TPO surpasses OPSD’s mean on all 13 splits while utilizing approximately 48% less GPU time. The computational cost is approximately one GPU-hour higher than GRPO per run (Table 1; App. C.9). P2TPO constructs these rewards without requiring an additional LLM judge, whereas CSPO reports an extra judging cost (§9).

AuthorsOpenReview ↗

Response to Reviewer HuN5

Part 2. Alignment, token masks, and independent audits

Alignment (W2/Q1)

Classical order-preserving sequence alignment is employed. Dynamic programming balances text similarity against skip costs to select an optimal one-to-one alignment, allowing weak matches to remain unpaired (App. B.1).

ItemRule
SimilarityS=2M/(np+nr)S=2M/(n_p+n_r); MM counts recursively matched characters in contiguous segments; prefix lengths np,nr≤192n_p,n_r\leq192
Pair / skip scoresS−0.6S-0.6 / −0.05-0.05 per skipped block
Pairing range∣j−k∣≤8\lvert j-k\rvert\leq8
Repeated contentGlobal path selects the correspondence
TiesSkip prediction → pair → skip reference
Reordering / split / mergeOrder-preserving, one-to-one matching; no explicit one-to-many or many-to-one matching; blocks may remain unmatched
MISSReference block with no assigned match

Token masks and omission routing (W2)

Character offsets are used to map blocks to tokens. Contiguous-token matching constructs SCR masks; negative coefficients emphasize errors, while positive coefficients emphasize correct tokens (App. B.2–B.3).

CaseAssignment
Matched tokensCorrect
Insertions / replacementsErroneous
DeletionsMark next token, or block-final token
Nearby errorsMerge spans separated by ≤3 non-error tokens
SCR eligibilityType-matched block, quality ≥0.8, nonzero coefficient
Ineligible / empty selected maskUniform weights
Credit preservationNonnegative, unit-mean weights preserve source-block sign and total before composition
MISS recipientFirst token of nearest right match in reference order; otherwise last token of nearest left match; otherwise final valid response token
MISS weight1; coefficients sharing a recipient are summed; bypass SCR

Independent audits (W2/Q1)

qwen3.8-max-xhigh is used to audit 240 eligible stratified correspondence pairs and 159 mask records from updates 1–10 (App. E.2.1; Tables 20–21).

MetricSample / countResult
Match precision463/477 unambiguous proposals97.06%
Match recall463/534 references with corresponding content86.70%
Wrong-reference proposals11/4772.31%
Spurious proposals3/4770.63%
Negative-mask macro precision79 records, including 12 with no annotated errors60.19%
Positive-mask macro precision80 records99.62%

These rules specify which blocks are compared, which tokens receive credit, and how unmatched references contribute. The results demonstrate the precision and stability of the methodology.


We appreciate your constructive suggestions regarding the continued-SFT comparison, fine-grained controls, and the alignment and token-mask methodologies. In Parts 1 and 2, we have provided the relevant performance comparisons, mechanism controls, and independent audits to address these questions. We welcome further questions and would be happy to clarify any remaining points.

Dear Reviewer HuN5,

Thank you very much for taking the time to review our paper. Your thoughtful feedback has been invaluable in improving our manuscript.

As the author–reviewer discussion period draws to a close, we would appreciate it if you could review our rebuttal and let us know whether our responses address your concerns. We would be pleased to clarify any remaining questions.

Thank you again for your time and consideration.

Best regards,

The Authors

Reviewer_3R59OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

Application: Convert a page of a scanned document to Markdown, using a trained policy with reference answers.

Background: Policy gradient methods such as REINFORCE and GRPO look at the rewards of different on-policy rollouts, and try to shift the policy to favor the higher-reward rollouts. The variance of the policy gradient signal can be quite high, in part because of the lack of credit assignment.

Approach: A bag of engineering tricks. The authors don't follow the policy gradient, but some other direction that is meant to improve components of the reward. It resembles GRPO: for each token of each rollout, they try to measure how much it "helped" or "hurt." Then they adjust the policy to increase the probabilities of the tokens that helped, and decrease the probabilities of the tokens that hurt.

A key input is a reference answer. The paper makes use of the reference answer not only to determine rewards but also to align rollouts to the reference and thus to each other. Then it can obtain normalized rewards within each aligned region, punish missing elements within an aligned region. When a rollout's region is similar enough to the reference region, token-level alignment is optionally used (the "SCR" method) to identify specific erroneous tokens and thus attribute credit and blame unevenly within the region. Finally, the reference answer is used to consider absolute as well as relative performance (but only when all rollouts have essentially identical scores).

Critical strengths

The paper is essentially about designing a token-by-token reward function. While the details are a bit hacky, they are motivated by a very reasonable-seeming engineering intuition -- namely, if there's a single correct answer and you know what it is, you should be able to do some kind of credit assignment to the individual tokens of a rollout answer.

The experiments show consistent gains, though they are small (about 1 point) and not tested for significance.

There are some nice ablations in section 4.3.

Critical weaknesses

The exposition is not ready for publication. This paper is mostly AI-written, and it is not reader-friendly. It is probably understandable by co-authors who are already oriented and have agreed on some terminology among themselves. But the authors need to actually write a paper that teaches the method, meaning that a human who reads it in order from start to finish should not be confused at any point.

The expository problems begin in section 2, whose subsections define a sequence of quantities -- using lots of jargon -- without ever saying how those quantities will be used. The actual policy update that uses them is buried in Appendix B.2.3! And Algorithms 1 and 3 are also buried in Appendix B and only referred to from there!

A lot of details are also underspecified. For example, in SCR, they don't say what alignment algorithm they use and how it interacts with tokenization, what counts as a “short gap,” or which neighboring tokens they penalize for a deletion.

The experimental comparisons are unfair because the hyperparameters were tuned for the proposed method and then the same hyperparameters were used for the baselines as well. This is inappropriate. In particular, the proposed method ensures higher variance of per-token updates, which may affect what learning rate works best. In other words, maybe the baseline methods just need to train for longer (on more rollouts) and/or train with a higher learning rate, since they don't do credit attribution?

Questions for authors

These are primarily meant as suggestions for a future submission, although I would also be interested in your answers if you care to give them.

  1. Parts of your method only work when you have a reference answer at training time, as I noted above. Since you have a reference answer, how about trying OPSD? Your setting actually seems like a very favorable one for OPSD, since tuning the model to agree better with reference Markdown is a much better-specified problem than tuning it to agree with reference CoT.

You would use a prompt saying something like "Here's the right answer. Now please write a transcription that agrees with it as much as possible without reordering elements or violating Markdown syntax. If you make a mistake that you can't recover from, just accept the error and keep going."

The alignment and attribution would then be handled for you automatically by the Transformer LM, rather than by your heuristics. And the teacher would also provide denser supervision than GRPO can, suggesting specific tokens at each position rather than just a reward for the particular token that the student sampled.

  1. You mention reordering, but does your alignment method allow reordering in the alignment? Or is it a standard diff-style method?

  2. Do you have any error analysis? I'd like to know what kinds of errors are corrected (and sometimes introduced) by your methods. Fig. 7-9 give a couple of examples, but 2 of 3 are correcting omissions, and it's not clear whether that would be the typical change of switching to your method. It might also be interesting to know whether the differences are more in particular types of region (text / formulas / tables) or particular types of document (long vs. short, or OmiDocBench categories).

  3. SCR usually hurts and only helps alongside SAP. Why?

  4. Your limitations section notes that the automatic alignment can be wrong -- have you evaluated its accuracy?

  5. The last 2 sentences of your limitations section mention other things you'd like to verify, including stat sig. Have you done so since submission?

9 comments
AuthorsOpenReview ↗

Response to Reviewer 3R59

Thanks very much for your detailed review and for spotting where the presentation of the method could have been improved. We certainly agree that the original manuscript would have benefited from a much clearer, more self-contained explanation of the method. Your comments have been very helpful in allowing us to improve the exposition and clarify the details of the implementation. Part 1 of this document details the revisions made as a result. Part 2 contains the additional experimental comparisons and the mechanism analysis.

Part 1. Revisions to writing and method presentation

ConcernRevisionLocation
W1: ReadabilityReorganize the motivation, component explanations, and question-driven evaluation; clarify terminology and visual presentation.Introduction, Methods, Experiments
W2: Main-text updateConnect token coefficients to the policy objective and training procedure.Section 2.5; Eqs. (3)–(4); Algorithm 1
W3; Q2: Implementation detailsSpecify alignment, token mapping, gap merging, and unmatched-block handling.Appendix B

1. Readability (W1)

We have substantially revised the manuscript to make the motivation, method, and experimental evidence easier to follow in a single reading. This revision covers the Introduction, the organization of the Methods and Experiments sections, terminology, figures, algorithms, and implementation details.

Introduction. It now begins with the background to the task: the value of full-page analysis using a single model. SFT is then introduced via a description of the training process and a discussion of the benefits (and indeed the necessity) of RL. The current state of the art and the challenges with RL are discussed, the observed phenomena are described, and the hypotheses that underpin the proposed algorithm are laid out. The section concludes with a summary of the key contributions.

Methods. We reorganized the section to provide an overview followed by a step-by-step explanation of the components. We have incorporated the component formulae into the figures, clarified the flow diagrams, and added an algorithm to connect these operations. In the component descriptions, we standardized terminology and introduced each new concept and its purpose before presenting details. Essential definitions are now in the main text; the appendix supplies detailed derivations. Readers can follow the complete method without repeatedly leaving the main text.

Experiments. We now begin with the motivation for the evaluation and five explicit research questions. The experimental setup follows, and the results are organized around those questions: overall effectiveness, credit-assignment granularity, component contributions, generality, and robustness to hyperparameters and training duration. This structure connects each experiment to the claim it tests and helps readers understand.

2. Main-text update (W2)

The revised Section 2.5 makes the connection between the defined quantities and the policy update explicit. Eq. (3) combines page feedback, SCR-weighted matched-block credit, and missing-block boundary feedback into the coefficient for each generated token:

+\omega\left[ \sum_{k\in\mathcal K_i}A_{i,k}^{\mathrm U}\mathbf1[t\in\mathcal B_{i,k}]d_{i,t} +\sum_{k\in\mathcal M_i}A_{i,k}^{\mathrm U}\mathbf1[t=t^\star(i,k)] \right]. \tag{3}$$ Eq. (4) then uses these coefficients in the clipped policy objective, where $\rho_{i,t}(\theta)$ is the probability ratio between the new and old policies: $$\widehat J(\theta)=\frac{1}{\sum_iT_i} \sum_i\sum_{t=1}^{T_i} \ell_{\mathrm{dual}} \left(\rho_{i,t}(\theta),A_{i,t}\right). \tag{4}$$ **Algorithm 1** brings sampling, alignment, credit computation, and policy updates together in the main text, showing how the components form an executable training procedure. ## 3. Alignment and SCR implementation (W3; Q2) We have clarified the alignment procedure and SCR’s token-level operations (**Appendix B**). - **Block alignment** uses monotonic, one-to-one banded dynamic programming and does not allow reordering. - **Within eligible matched blocks**, SCR recursively matches the longest shared contiguous token-piece sequences. - **Generated-token character offsets** map blocks to token intervals; reference tokenization excludes special tokens and removes outer formula delimiters. - **Error spans** separated by at most three non-error tokens are merged. - **Wholly unmatched reference blocks** are handled separately through boundary proxies. These additions specify the alignment, tokenization, gap threshold, and deletion targets the reviewer requested.
AuthorsOpenReview ↗

Response to Reviewer 3R59

Part 2. Experimental evidence and mechanism analysis

1. Learning rates and training budgets (W4)

The gains persist with method-specific learning-rate selection and longer training. We swept the same four rates separately for each method; all six select 1e−6, which corresponds to the same GRPO base in theory.

Method2e−75e−71e−62e−6
GRPO82.4482.7483.8781.38
GSPO82.0582.2683.4682.67
GDPO82.5382.7783.5382.09
RDPO82.5182.7483.5382.68
P2BPO82.6783.8984.2583.24
P2TPO82.6483.8484.6583.80

In a separate 50K-SFT study with matched rollout budgets, 500 updates yield OmniDocBench scores of 91.49 for P2TPO versus 90.42 for GDPO, the strongest response-level RL baseline (Table 15). Thus, increasing the learning rate or training duration does not close the gap in these tested settings.

2. Reference-conditioned distillation (Q1)

We added an OPSD variant using a reference-conditioned frozen teacher. The teacher sees reference Markdown, then scores student-generated tokens; training uses sampled-token reverse KL. With matched initialization, data, and rollout budgets, its OmniDocBench score is 92.31 versus P2TPO’s 93.12 (Table 1). OPSD uses a stability-selected learning rate of 5e−8.

OPSD exploration. We found OPSD unstable in our setting, with performance consistently declining across all four learning rates in the original-version sweep. We therefore conducted over 60 training runs, exploring multiple samples per prompt, prompt refinements, a larger learning-rate range extending down to 1e−8, and both full-distribution reverse KL and sampled-token KL objectives. The reported configuration achieved the best improvement during this exploration.

These experiments also confirm the value of your idea. If we can design a solution that addresses instability and enables the model to provide more reliable token supervision for prefix generation by teachers who have not undergone SFT, this is a direction worth exploring.

3. Why SCR benefits from UAP (Q4; W4)

UAP can change the sign of block credit and therefore which tokens SCR emphasizes. UAP (“SAP” in the question) includes missing outcomes in the comparison. A matched block that receives negative credit when compared only with other matched blocks can receive positive credit once missing outcomes are included. SCR then emphasizes correct tokens rather than error spans.

In a frozen-output diagnostic, 108/447 initially nonzero matched-only coefficients in groups containing missing outcomes switch from negative to positive. The performance comparisons are consistent with this mechanism (Appendix C.3; Table 3).

ComparisonSplits with higher mean
Adding SCR with UAP11/13
Adding SCR without UAP2/13
Aligned SCR vs. shifted placement10/13
Aligned SCR vs. randomized placement12/13

Placement controls preserve each block’s coefficient multiset, allowing informed placement beyond coefficient dispersion.

4. Alignment accuracy (Q5)

Block correspondence is accurate. Audits use blinded independent-model annotations (Tables 20–21).

AuditSamplePrecisionRecall
Block matching240 stratified pairs97.06%86.70%
Negative mask79 records60.19%—
Positive mask80 records99.62%—

5. Corrected and introduced errors (Q3)

Improvements go beyond omissions and include identifiable regressions. Improved records have a normalized edit-distance (NED) decrease ≥0.02 (Appendix E.1).

AnalysisFinding
Non-omission share of improved records69%
3,000–10,000 reference characters/pageSummed NED −9.47%
<1,000 reference characters/pageChange CI includes zero
Research-paper categorySummed NED +2.15%

Formatting and merge/split cases both improve and regress. Text contributes most to reduced summed NED; tables/formulas and books also improve.

6. Significance and remaining checks (Q6)

All six aggregate gains are significant after Holm correction. Two-sided Welch tests use five run-level means per method. (Table 1 results; Appendix C.1.4).

BaselineP2TPO gain (points)95% CIHolm p
GRPO+0.78[0.23, 1.33]<0.05
GSPO+1.19[0.66, 1.72]<0.01
GDPO+1.12[0.58, 1.67]<0.01
RDPO+1.12[0.57, 1.67]<0.01
cSFT+3.14[2.61, 3.67]<0.001
OPSD variant+4.46[3.93, 4.98]<0.001

We have performed substantial revisions and conducted additional experiments to tackle your concerns and clarify the motivation, method, and evidence. We sincerely hope these efforts provide a stronger basis for your assessment of our work, and we welcome further questions and suggestions.

Reviewer_3R59OpenReview ↗

I didn't list all 99 weaknesses ...

  1. Readability (W1)

Right, my main concern was overall readability. I'll try to give the revised version a fresh read, as I said here.

  1. Main-text update (W2) ...
  2. Alignment and SCR implementation (W3; Q2) ...

These respond to the 2nd and 3rd paragraphs of the "weaknesses" section. But those paragraphs were not meant as a complete list of weaknesses. They were just examples of how the presentation was too disorganized and incomplete to accept. So, I should just read the new version.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to Reviewer qRCV

We sincerely appreciate your thoughtful review and your recognition of the strengths in reference-element granularity, UAP's approach to omissions, and our evaluation. Thank you as well for your positive comments on placement controls and implementation overhead. Below, we clarify the alignment procedure and present the relevant comparisons and analyses.

Our response has two parts: Part 1 covers alignment and comparisons (W1/Q1, W2); Part 2 addresses recipient accuracy, omission recovery, and reliability (W3–W4/Q2).

Part 1: Alignment and Controlled Comparisons

Alignment and SCR implementation (W1/Q1): Appendix B details how reference blocks are matched and how their feedback reaches generated tokens.

  • Block alignment uses text similarity and monotonic, one-to-one banded dynamic programming. Fixed tie-breaking rules handle repeated text, while reordered or resegmented equivalents may remain unmatched (App. B.1).
  • Token alignment uses character offsets to map generated blocks to token intervals and recursively matches the longest shared contiguous token-piece sequences. SCR applies to type-matched blocks with quality at least 0.8 and nonzero coefficients.
  • Error spans include insertions and replacements. Deletions mark the next generated token, or the block's last token for trailing gaps. Spans separated by at most three non-error tokens are merged, including the intervening positions.
  • SCR weighting emphasizes merged error spans for negative coefficients and remaining correct positions for positive coefficients. Appendix B.3 specifies token tie-breaking and uniform-weight fallbacks.
  • Unmatched references route their coefficients with weight 1 to the nearest right match's first token in reference order. If no right match exists, they use the nearest left match's last token; if neither exists, the final valid response token. Contributions sharing a proxy add and bypass SCR (App. B.2).

Controlled comparisons (W2): We separate routing granularity from imperfect-consensus calibration.

Routing granularity. Blocks of the same type can improve and worsen within a single response; a shared type coefficient cannot capture all these differences. Table 11 compares block and type routing with local feedback only and UAP/SCR disabled, using the same table correspondences and TEDS scores. This is a full-page granularity control motivated by CSPO's attribute-based routing, rather than a reproduction of its table-attribute method.

Selected Table 11 results: Overall, five-run mean ± SD.

Benchmark / splitType routingBlock routing
OmniDocBench v1.692.39 ± 0.2892.46 ± 0.26
PureDocBench Clean82.94 ± 0.6083.40 ± 0.24
PureDocBench Digital79.34 ± 0.6379.78 ± 0.09
PureDocBench Real68.16 ± 0.8868.42 ± 0.38

Block routing achieves higher means on 8/13 splits. Adding page feedback to block routing improves over both pure-local variants on 13/13 splits, supporting complementary page and block signals.

Quality-gap calibration. Centering identical outcomes removes their absolute quality information. Under consensus at quality r<1r<1, UAP assigns −λ(1−r)-\lambda(1-r), preserving the gap to perfect quality. The NGRPO-inspired control appends a virtual maximum-quality outcome before normalization; its coefficient becomes approximately gap-independent when the stabilizer is negligible relative to the gap-dependent denominator term. Only UAP's low-dispersion branch changes; alignment, missing-outcome zeros, omission routing, SCR, and page feedback remain fixed (App. C.3.1).

Selected Table 3 results: Overall, five-run mean ± SD.

Benchmark / splitNGRPO calibrationFull P2TPO
OmniDocBench v1.692.90 ± 0.1593.12 ± 0.23
PureDocBench Clean83.93 ± 0.2484.68 ± 0.74
PureDocBench Digital80.32 ± 0.2581.03 ± 0.60
PureDocBench Real69.42 ± 0.2770.33 ± 0.70

Full P2TPO achieves higher means on 9/13 splits, supporting UAP's quality-gap-dependent penalty within this controlled setup.

Feedback placement and conservation. Table 3 also tests whether SCR benefits from where feedback is placed. Random support and circular shift preserve coefficient values and selected-token counts while changing their positions.

Placement controlSplits with higher mean under aligned SCR
Random support12/13
Circular shift10/13

These controls support alignment-informed feedback beyond nonuniform weighting alone. Separately, conserving the block credit budget yields higher means than the non-conserving control on 13/13 splits, supporting the role of credit conservation.

AuthorsOpenReview ↗

Response to Reviewer qRCV

Part 2: Recipient Accuracy, Omission Recovery, and Reliability

Recipient correctness and omissions (W3/Q2): Conservation determines the allocation of the credit budget, while independent annotations evaluate its distribution. For eligible blocks with a nonempty, non-full mask, SCR increases the target's share of absolute coefficient mass precisely when redistribution is active, and mask precision surpasses the target's fraction of block tokens. (App. D.3).

The qwen3.8-max-xhigh audit encompasses 240 eligible stratified pairs from training updates 1 to 10 (Table 20):

Matching metricCountEstimate95% page-bootstrap CI
Precision463/47797.06%[95.26%, 98.57%]
Recall463/53486.70%[82.20%, 90.68%]
Incorrect-reference proposals11/4772.31%[0.98%, 3.94%]
Spurious proposals3/4770.63%[0.00%, 1.43%]

These results support the effectiveness of our block matching.

Table 21 presents an evaluation of token masks:

Mask targetRecordsMacro precisionMedian
Errors, negative coefficient7960.19%80.00%
Correct positions, positive coefficient8099.62%100.00%

All 79 negative-mask records are included, with 12 exhibiting no annotated errors and zero precision. Matching and correct-region selection demonstrate accuracy in this audit.

Excluding only omission routing, while retaining missing-outcome zeros and other components, reduces the mean Overall metric in 11/13 splits (Table 3):

Selected Table 3 results: five-run meanWithout omission routingFull P2TPO
OmniDocBench v1.692.8693.12
PureDocBench Clean83.7484.68
PureDocBench Real69.6770.33
FinixDocBench70.0570.92

The separate paired GRPO–P2TPO output analysis utilizes the same seed after 100 updates on OmniDocBench v1.6, comprising 1,641 pages and 5,142 reference-anchored records. Table 17 reports:

GRPO-side error categoryRecordsImprovedWorsened
Omission39627.53%0.00%
Merge / split1,5646.59%6.65%
Format change / minor normalization1,4095.25%4.68%

Improvement or worsening is defined as a normalized edit distance (NED) decrease or increase of at least 0.02. Of 396 omission records, 109 show improvement; omission NED begins at its upper bound of 1, which precludes worsening. Table 18 presents lower summed error across all three element types:

RegionImproved recordsWorsened recordsNet NED reduction
Text268287+52.55
Table7760+12.97
Formula71+3.63

The ablation analysis supports the contribution of omission routing during training. Paired outputs indicate reduced omission severity.

Variability and paired reliability (W4/Q2): Page-level and training-run analyses address different sources of variation. For the same seed pair, Table 19 reports:

Reference blocks/pagePagesRelative reduction in summed NED95% page-bootstrap CI
≤41,302−1.30%[−4.50%, 2.21%]
5–9111+8.62%[−1.53%, 18.16%]
10–1971+11.13%[−2.07%, 24.39%]
≥20157+7.39%[2.31%, 14.44%]

Medium- and high-density groups exhibit larger estimated reductions, consistent with preserving local feedback across multiple elements.

Each run contributes one equal-weight mean over 13 splits. P2TPO averages 84.65. Table 1 and App. C.1.4 report two-sided Welch comparisons using five runs:

BaselineAggregate mean95% CI: P2TPO − baselineHolm-adjusted p
GRPO83.87[0.23, 1.33]<0.05
GSPO83.46[0.66, 1.72]<0.01
GDPO83.53[0.58, 1.67]<0.01
RDPO83.53[0.57, 1.67]<0.01
cSFT81.51[2.61, 3.67]<0.001
OPSD80.19[3.93, 4.98]<0.001

Confidence intervals are pointwise, and Holm correction applies to all six tests. All results remain significant, supporting an aggregate advantage across training runs (+0.78 points over GRPO).


Together, the controlled comparisons, alignment audits, and error analyses support reference-level feedback and its contribution to parsing performance. Thank you again for your thoughtful review. We welcome further questions and discussion.

Dear Reviewer qRCV,

Thank you very much for taking the time to review our paper and for your positive comments. Your thoughtful feedback has been invaluable in improving our manuscript.

As the author–reviewer discussion period concludes, we kindly ask if you could review our rebuttal and let us know whether our responses address your concerns. We would be pleased to clarify any remaining questions.

Thank you once again for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.