- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes a unified fixed-batch autoregressive update template (Sections 1–2, Eqs. 1–2, Table 1) in which language-model pre-training (CE) and reinforcement-learning post-training (PG, PPO, GRPO, DAPO) differ in their trajectory distribution , detached per-token learning weight , and normalization . Building on this view, Section 3 introduces UniCredit, a lightweight (1.06M parameters at 7B scale; 0.30M in pre-training) causal credit decoder trained on detached final-layer hidden states (Eq. 3) to predict token-level (Eqs. 15–16). In pre-training (Section 3.2, Eqs. 4–8), the head learns multi-scale surprisal targets () mixed into cross-entropy with weight . In post-training (Sections 3.3–3.6, Eqs. 9–14), it combines masked GAE (Eqs. 9–10), explicit self-assessment and semantic-chunk suffix labels (Eqs. 11, 20), an outcome-preserving token advantage residual with (Eq. 13), and readiness/stability guards that fall back to DAPO when credit quality is low (Eq. 14).
Empirically, matched FineWeb-Edu pre-training runs at 455M (20.0B nominal / 20.97B exact tokens) and 1.5B (50.0B tokens) improve the 6-task zero-shot mean from 42.80 to 43.58 (+0.78) and from 52.54 to 53.23 (+0.69, Table 3). For post-training on Qwen2.5-Math-7B (Section 5.2, Table 4), the paper reports a preliminary AIME-2024 avg@32 increase from 36.0% (DAPO) to 37.8% (UniCredit), alongside a cross-domain summary table (Table 5) and safety/curriculum diagnostics (Figure 3, Appendix C Table 6 and Figure 4). While the fixed-batch conceptual interface, explicit gradient boundaries, and candid negative diagnostics (e.g., collapse under an unsafe 80% learned share) are valuable, the submission reads like an incomplete work-in-progress: the core RL post-training and cross-domain experiments are explicitly preliminary or not yet run under matched budgets, the 37.8% vs. 36.0% AIME comparison is confounded by unequal training steps (step 150 vs. step 100) and a 0% effective policy share in the guarded configuration (Table 6), pre-training omits the direct baseline of weighting CE with the already-computed online target without a learned head, and there are severe mathematical, specification, and cross-referencing defects throughout the manuscript.
Critical strengths
1. Clean fixed-batch update taxonomy and outcome-preserving token advantage design.
Table 1 provides a tidy unified notation mapping pre-training CE and RL post-training methods (PG, PPO, GRPO, DAPO) onto trajectory distribution , detached token weight , and normalization . In addition, decomposing token advantages as with zero-mean residuals and bounded scale (Section 3.5, Eq. 13) is a sensible mechanism to prevent within-sequence normalization from erasing the trajectory's overall outcome sign .
2. Well-controlled pre-training experiments at two scales and candid reporting of RL failure modes.
The FineWeb-Edu pre-training comparison (Section 5.1, Table 3) matches architecture (455M and 1.5B), tokenizer, data order, and learning-rate schedule, showing consistent positive improvements on 11 of 12 zero-shot evaluations (+0.78 and +0.69 mean points). In post-training, the authors transparently document that an unguarded 80% learned-credit mixture collapses (Reward -1, length approx. 31, in Table 6) and that terminal self-score correlation is near zero (0.018 in Figure 4).
Critical weaknesses
1. The post-training and cross-domain empirical claims rely on unfinished, unmatched, or step-confounded experiments.
- Unfinished and step-confounded 7B math RL (Table 4, Figure 2, Table 9): As detailed under Soundness, the
37.8%vs.36.0%AIME-2024result compares step 150 (after a 100-step DAPO warm-up) against step 100 of DAPO on a single 30-problem benchmark (where a1.8pp difference across samples is only 17 responses, well within binomial sampling noise pp), while Table 6 indicates that the adaptive guarded configuration actually had a0%effective policy share (Stable DAPO fallback). - Unmatched external numbers presented as a cross-domain comparison (Table 5 & Appendix G): Section 5.3 (Table 5, p. 9) presents cross-domain gains on
LiveCodeBench v2(29.5vs.28.6),LeetCode(34.0vs.33.3),IFEval strict(77.5vs.73.38), andFollowBench hard(+5.0vs.+4.0). However, Appendix G (p. 15, lines 791–802) admits that the code baselines are copied from an external paper (PRIME, using a different base model, dataset, and budget), theIFEvalbaseline is an untunedQwen2.5-7B-Instructcheckpoint, and matched DAPO/GRPO runs have not been executed yet ("We will train DAPO/GRPO and UNICREDIT from the same checkpoint...", line 800). - Unmeasured efficiency claims (Table 8 & Appendix E): Table 8 (p. 14) reports only a target throughput ( tok/s/GPU) for UniCredit without a single measured throughput, memory, or wall-clock number for the proposed method.
2. Missing causal ablations for the learned credit decoder and architectural disconnect between pre-training and post-training.
- Pre-training omits the direct-target baseline and the unified causal decoder block: Because the multi-scale surprisal target in Eq. (5) is already computed online from the current policy's gold-token log-probabilities on every training batch, the paper never tests the most direct causal baseline: weighting the pre-training CE loss directly with the already-computed target (via Eqs. 7–8) without training the auxiliary predictor . Moreover, although Section 3.1 (Eq. 3) pitches a unified causal credit decoder with a causal attention block predicting , line 157 reveals that pre-training actually uses only
0.30M-parameter linear advantage/importance readouts—omitting the reward head, value head, and causal attention block altogether—and the1.5Bpre-training architecture is never specified, nor are the pre-trained Qwen3-style models ever used for RL post-training (which switches toQwen2.5-Math-7B). - Post-training lacks component ablations: Figure 3 (p. 9) is explicitly acknowledged not to be a causal ablation (line 465), and there are no controlled ablations isolating the four heads (), exact ratio vs. predicted , or Eq. (13)'s outcome-preserving residual against standard GAE/DAPO at matched steps.
Questions for authors
-
Assessment-critical (Exact run lineage of the
37.8%AIME result, step-150 DAPO score, and actual schedule):- How was the
37.8%checkpoint in Table 4 / Table 9 produced, and why does Figure 2 show a separateUNICREDIT (preliminary)curve that is around33.5%at step 100 whereas Table 9 states that steps0–100(36.0%at step 100) are the shared DAPO warm-up checkpoints? What is pure DAPO'savg@32at step 150 (omitted from the list in Appendix F line 778)? - Since Table 6 reports a
0%policy share (Stable DAPO fallback) forAdaptive, guarded, what was the actual per-step value of and the explicit self-score weight in the run that reached37.8%?
- How was the
-
Assessment-critical (Actor gradient formulation in Eq. 14 and the undefined
0.610 corr.in Table 6):- Given that Eq. (16) detaches both and , what is the exact differentiable actor loss corresponding to in Eq. (14) (and how is the explicit self-score branch incorporated)? Why predict from when exact old log-probabilities are already stored?
- In Table 6 (Row 4,
Pre-training, 40k), how isImp. valid = 0.610 corr.computed when the pre-training importance target is the constant (Section 3.2, line 175), for which Pearson correlation is undefined?
-
Assessment-critical (Direct-target pre-training baseline, completed matched RL runs, and measured throughput):
- In
455Mand1.5Bpre-training, what is the zero-shot performance if you apply Eqs. (7)–(8) directly to the online multi-scale surprisal target (Eq. 5) without the auxiliary prediction head? - Can you provide the measured tokens/s/GPU throughput, stage-wise timing breakdown, and peak memory of UniCredit (promised in Appendix E line 759 but missing from Table 8), along with the 1.5B model architecture specifications and all missing hyperparameters from Section 3?
- In