acceptodds
Under review as a conference paper at ICLR 2027

Unifying Language Model Pre-Training and Reinforcement Post-Training with Learned Token Credit

Abstract

Pre-training and reinforcement post-training are usually treated as different algorithms. We show that both optimize one weighted autoregressive objective, differing only in the trajectory distribution and per-token learning value. This view motivates \method, a lightweight causal credit decoder inside the policy. From detached final hidden states, it learns reward, value, advantage, and importance targets; rule functions and held-out semantic chunks provide explicit supervision. No policy-sized reward model or critic is required. On matched FineWeb-Edu runs, improves the six-task zero-shot mean from 42.80 to 43.58 at 455M parameters and from 52.54 to 53.23 at 1.5B. We further reproduce a Qwen2.5-Math-7B DAPO baseline reaching 36.0% AIME-2024 avg@32 and study a preliminary trajectory reaching 37.8%. Diagnostics show why calibration matters: importance validity reaches roughly 91%, while advantage and verbal self-score correlations remain modest. The framework therefore unifies generation and judgment in one policy, while admitting learned credit only after its predictive quality is demonstrated.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Pretraining, Reinforcement Learning, Training paradigms

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QFu1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a unified fixed-batch autoregressive update template (Sections 1–2, Eqs. 1–2, Table 1) in which language-model pre-training (CE) and reinforcement-learning post-training (PG, PPO, GRPO, DAPO) differ in their trajectory distribution qq, detached per-token learning weight ωt\omega _ t, and normalization Z(τ)Z(\tau). Building on this view, Section 3 introduces UniCredit, a lightweight (1.06M parameters at 7B scale; 0.30M in pre-training) causal credit decoder trained on detached final-layer hidden states sg[ht]\mathrm{sg}[h _ t] (Eq. 3) to predict token-level (r^t,V^t,A^t,ρ^t)(\hat{r} _ t, \hat{V} _ t, \hat{A} _ t, \hat{\rho} _ t) (Eqs. 15–16). In pre-training (Section 3.2, Eqs. 4–8), the head learns multi-scale surprisal targets (K={T,32,128,512}\mathcal{K} = \lbrace T, 32, 128, 512 \rbrace) mixed into cross-entropy with weight α=0.05\alpha = 0.05. In post-training (Sections 3.3–3.6, Eqs. 9–14), it combines masked GAE (Eqs. 9–10), explicit self-assessment and semantic-chunk suffix labels (Eqs. 11, 20), an outcome-preserving token advantage residual Ai,ttoken=Gi+ciui,tA _ {i,t}^{\mathrm{token}} = G _ i + c _ i u _ {i,t} with meantui,t=0\mathrm{mean} _ t u _ {i,t} = 0 (Eq. 13), and readiness/stability guards that fall back to DAPO when credit quality is low (Eq. 14).

Empirically, matched FineWeb-Edu pre-training runs at 455M (20.0B nominal / 20.97B exact tokens) and 1.5B (50.0B tokens) improve the 6-task zero-shot mean from 42.80 to 43.58 (+0.78) and from 52.54 to 53.23 (+0.69, Table 3). For post-training on Qwen2.5-Math-7B (Section 5.2, Table 4), the paper reports a preliminary AIME-2024 avg@32 increase from 36.0% (DAPO) to 37.8% (UniCredit), alongside a cross-domain summary table (Table 5) and safety/curriculum diagnostics (Figure 3, Appendix C Table 6 and Figure 4). While the fixed-batch conceptual interface, explicit gradient boundaries, and candid negative diagnostics (e.g., collapse under an unsafe 80% learned share) are valuable, the submission reads like an incomplete work-in-progress: the core RL post-training and cross-domain experiments are explicitly preliminary or not yet run under matched budgets, the 37.8% vs. 36.0% AIME comparison is confounded by unequal training steps (step 150 vs. step 100) and a 0% effective policy share in the guarded configuration (Table 6), pre-training omits the direct baseline of weighting CE with the already-computed online target At∗A _ t^* without a learned head, and there are severe mathematical, specification, and cross-referencing defects throughout the manuscript.

Critical strengths

1. Clean fixed-batch update taxonomy and outcome-preserving token advantage design.
Table 1 provides a tidy unified notation mapping pre-training CE and RL post-training methods (PG, PPO, GRPO, DAPO) onto trajectory distribution qq, detached token weight ωt\omega _ t, and normalization Z(τ)Z(\tau). In addition, decomposing token advantages as Ai,ttoken=Gi+ciui,tA _ {i,t}^{\mathrm{token}} = G _ i + c _ i u _ {i,t} with zero-mean residuals meantui,t=0\mathrm{mean} _ t u _ {i,t} = 0 and bounded scale cic _ i (Section 3.5, Eq. 13) is a sensible mechanism to prevent within-sequence normalization from erasing the trajectory's overall outcome sign GiG _ i.

2. Well-controlled pre-training experiments at two scales and candid reporting of RL failure modes.
The FineWeb-Edu pre-training comparison (Section 5.1, Table 3) matches architecture (455M and 1.5B), tokenizer, data order, and learning-rate schedule, showing consistent positive improvements on 11 of 12 zero-shot evaluations (+0.78 and +0.69 mean points). In post-training, the authors transparently document that an unguarded 80% learned-credit mixture collapses (Reward -1, length approx. 31, KL>2\mathrm{KL} \gt 2 in Table 6) and that terminal self-score correlation is near zero (0.018 in Figure 4).

Critical weaknesses

1. The post-training and cross-domain empirical claims rely on unfinished, unmatched, or step-confounded experiments.

  • Unfinished and step-confounded 7B math RL (Table 4, Figure 2, Table 9): As detailed under Soundness, the 37.8% vs. 36.0% AIME-2024 result compares step 150 (after a 100-step DAPO warm-up) against step 100 of DAPO on a single 30-problem benchmark (where a 1.8 pp difference across 30×32=96030 \times 32 = 960 samples is only 17 responses, well within binomial sampling noise SE≈2.2\mathrm{SE} \approx 2.2 pp), while Table 6 indicates that the adaptive guarded configuration actually had a 0% effective policy share (Stable DAPO fallback).
  • Unmatched external numbers presented as a cross-domain comparison (Table 5 & Appendix G): Section 5.3 (Table 5, p. 9) presents cross-domain gains on LiveCodeBench v2 (29.5 vs. 28.6), LeetCode (34.0 vs. 33.3), IFEval strict (77.5 vs. 73.38), and FollowBench hard (+5.0 vs. +4.0). However, Appendix G (p. 15, lines 791–802) admits that the code baselines are copied from an external paper (PRIME, using a different base model, dataset, and budget), the IFEval baseline is an untuned Qwen2.5-7B-Instruct checkpoint, and matched DAPO/GRPO runs have not been executed yet ("We will train DAPO/GRPO and UNICREDIT from the same checkpoint...", line 800).
  • Unmeasured efficiency claims (Table 8 & Appendix E): Table 8 (p. 14) reports only a target throughput (≥1140\ge 1140 tok/s/GPU) for UniCredit without a single measured throughput, memory, or wall-clock number for the proposed method.

2. Missing causal ablations for the learned credit decoder and architectural disconnect between pre-training and post-training.

  • Pre-training omits the direct-target baseline and the unified causal decoder block: Because the multi-scale surprisal target At∗A _ t^* in Eq. (5) is already computed online from the current policy's gold-token log-probabilities st=log⁡pθ(yt∣y<t)s _ t = \log p _ \theta(y _ t \mid y _ {\lt t}) on every training batch, the paper never tests the most direct causal baseline: weighting the pre-training CE loss directly with the already-computed target At∗A _ t^* (via Eqs. 7–8) without training the auxiliary predictor ϕ\phi. Moreover, although Section 3.1 (Eq. 3) pitches a unified causal credit decoder with a causal attention block predicting (r^t,V^t,A^t,ρ^t)(\hat{r} _ t, \hat{V} _ t, \hat{A} _ t, \hat{\rho} _ t), line 157 reveals that pre-training actually uses only 0.30M-parameter linear advantage/importance readouts—omitting the reward head, value head, and causal attention block altogether—and the 1.5B pre-training architecture is never specified, nor are the pre-trained Qwen3-style models ever used for RL post-training (which switches to Qwen2.5-Math-7B).
  • Post-training lacks component ablations: Figure 3 (p. 9) is explicitly acknowledged not to be a causal ablation (line 465), and there are no controlled ablations isolating the four heads (r^,V^,A^,ρ^\hat{r}, \hat{V}, \hat{A}, \hat{\rho}), exact ratio ρt\rho _ t vs. predicted ρ^i,t\hat{\rho} _ {i,t}, or Eq. (13)'s outcome-preserving residual against standard GAE/DAPO at matched steps.
Questions for authors
  1. Assessment-critical (Exact run lineage of the 37.8% AIME result, step-150 DAPO score, and actual β\beta schedule):

    • How was the 37.8% checkpoint in Table 4 / Table 9 produced, and why does Figure 2 show a separate UNICREDIT (preliminary) curve that is around 33.5% at step 100 whereas Table 9 states that steps 0–100 (36.0% at step 100) are the shared DAPO warm-up checkpoints? What is pure DAPO's avg@32 at step 150 (omitted from the list in Appendix F line 778)?
    • Since Table 6 reports a 0% policy share (Stable DAPO fallback) for Adaptive, guarded, what was the actual per-step value of β\beta and the explicit self-score weight in the run that reached 37.8%?
  2. Assessment-critical (Actor gradient formulation in Eq. 14 and the undefined 0.610 corr. in Table 6):

    • Given that Eq. (16) detaches both A^i,t\hat{A} _ {i,t} and ρ^i,t\hat{\rho} _ {i,t}, what is the exact differentiable actor loss corresponding to JDAPO(A^i,t,ρ^i,t)J _ {\mathrm{DAPO}}(\hat{A} _ {i,t}, \hat{\rho} _ {i,t}) in Eq. (14) (and how is the explicit self-score branch incorporated)? Why predict ρ^i,t\hat{\rho} _ {i,t} from sg[ht]\mathrm{sg}[h _ t] when exact old log-probabilities log⁡πold\log \pi _ {\mathrm{old}} are already stored?
    • In Table 6 (Row 4, Pre-training, 40k), how is Imp. valid = 0.610 corr. computed when the pre-training importance target is the constant ρt∗≡1\rho _ t^* \equiv 1 (Section 3.2, line 175), for which Pearson correlation is undefined?
  3. Assessment-critical (Direct-target pre-training baseline, completed matched RL runs, and measured throughput):

    • In 455M and 1.5B pre-training, what is the zero-shot performance if you apply Eqs. (7)–(8) directly to the online multi-scale surprisal target At∗A _ t^* (Eq. 5) without the auxiliary prediction head?
    • Can you provide the measured tokens/s/GPU throughput, stage-wise timing breakdown, and peak memory of UniCredit (promised in Appendix E line 759 but missing from Table 8), along with the 1.5B model architecture specifications and all missing hyperparameters from Section 3?
Reviewer_Ab2jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UNICREDIT presents a common weighted-likelihood formulation for language-model pretraining and post-training. A small causal head predicts token-level training signals from detached representations, with teacher signals and readiness gates controlling its contribution. The paper reports matched pretraining experiments and preliminary post-training results.

Critical strengths
  1. A shared interface for token-level training signals is a useful systems idea. Detaching the head inputs and gating unreliable predictions are sensible safeguards.
  2. The matched pretraining experiments provide concrete evidence: the reported mean benchmark scores improve from 42.80 to 43.58 at 455M and from 52.54 to 53.23 at 1.5B under the stated budgets.
Critical weaknesses
  1. The post-training evidence is incomplete for the scope of the claims. The main AIME result and broader comparisons are marked preliminary, and several external reference results are not matched experiments. This makes the claimed unification across training stages difficult to assess empirically.
  2. The benefit of the learned credit heads is not sufficiently isolated. The guarded setting can reduce to the underlying DAPO update when the effective head share is zero. The early curriculum results do not yet establish a causal advantage from active learned heads over a matched zero-share control.
Questions for authors
  1. Can the authors provide completed, matched post-training results, including training variability and actual throughput rather than throughput targets?
  2. Under the same budget, how do active heads compare with zero-share heads and direct teacher-signal weighting? Please report the effective head contribution over training.
Reviewer_fVppOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents UniCredit, a lightweight decoder that predicts token-level training weights from detached language-model hidden states. It expresses pre-training and reinforcement post-training through a common weighted log-likelihood interface, with regime-specific supervision and safeguards controlling learned credit. Matched pre-training experiments report six-task average improvements of 0.78 and 0.69 percentage points at 455M and 1.5B parameters. Post-training results suggest a preliminary AIME improvement over DAPO.

Critical strengths

(1) The practical motivation is compelling: estimating dense credit using a small auxiliary module could improve training without maintaining another policy-sized model. (2) The pre-training comparisons control initialization, data order, optimizer, and token budget. Improvements across two scales provide encouraging evidence. (3) The organization is generally clear, and the distinction between local gradient equivalence and identical optimization dynamics is appropriately acknowledged in Appendix B.

Critical weaknesses

(1)Equation (14) substitutes predicted importance ratios into a DAPO surrogate, while Equations (3) and (16) detach these predictions from the actor. Under the standard PPO interpretation, a surrogate containing only detached ratios and advantages has zero actor gradient. If the implementation instead uses a differentiable exact ratio or an explicit log-probability factor, that computational graph must be specified. This affects the central claim that learned credit updates the policy. (2) Identical-budget reruns, multiple seeds, and important ablations remain unfinished. Comparing UniCredit at step 150 with DAPO’s best checkpoint at step 100 needs a clearly defined common budget and selection protocol. The active-head curriculum cannot isolate gains from ordinary DAPO learning, while the guarded configuration in Table 6 contributes no learned policy signal. Cross-domain comparisons include external references, and the reported throughput threshold is a target rather than a measured UniCredit result. (3) The manuscript mixes completed results, preliminary measurements, and future acceptance criteria. Section 3.4 and Appendix D differ over whether soft mathematical similarity enters the primary reward. “Appendix 4” is an incorrect reference, and “statistically flat” is unsupported without statistical analysis. The engineering contribution is potentially useful, but broad claims of unification exceed the demonstrated advance.

Questions for authors

(1) What exact differentiable expression produces the learned branch’s actor gradient? (2) How are chunk-length weighting, residual bounds, and readiness thresholds implemented? (3) Can completed, compute-matched multi-seed results establish improvement while reporting actual head activation and overhead? (4) Does UniCredit outperform direct teacher weighting and exact-ratio alternatives?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.