acceptodds
Withdrawn from ICLR 2027

PACT: From Credit Assignment to Critic Alignment

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning has become a central component of large language model (LLM) post-training, yet token-level credit lacks a generally accepted mathematical definition, leaving its relationship to commonly used training signals unclear. We formulate three regularity conditions, namely Completeness, Prefix Consistency, and Neutrality, and prove that they uniquely determine token-level credit. This characterization provides a unified basis for explaining phenomena across existing algorithms and guides the development of an improved actor-critic training procedure. Through this lens, an ideal teacher in On-Policy Distillation (OPD) acts as an implicit critic, yielding an expected policy gradient proportional to that induced by token-level credit. Response-level REINFORCE Leave-One-Out (RLOO) signals match the expected policy-gradient contribution of token-level credit despite their coarser granularity. We further establish approximate credit sparsity under bounded outcome rewards and show how intermediate critic errors in Generalized Advantage Estimation (GAE) can become comparable to the underlying credit. These motivate Policy Aligned Critic Training (PACT), which adopts an Actor-then-Critic update order to apply importance sampling correction to critic training and better align the critic with the updated policy. In agentic mathematical reasoning, PACT achieves 72.87% average accuracy across four benchmarks, outperforming GRPO and PPO by 8.80 and 13.16 percentage points, respectively. On SWE-bench Verified, PACT achieves a pass rate of 67.4%, outperforming PPO, GRPO, and SAO by 2.4, 2.0, and 3.8 percentage points, respectively.

Primary area:
reinforcement learning
Keywords:
Reinforecement Learning, Credit Assignment, Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_4wfLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PACT, which improves critic training through Actor-then-Critic updates, importance-corrected targets, and BCE. The authors derive token-level credit under three regularity conditions and report performance gains on mathematical reasoning tasks and SWE-bench Verified.

Critical strengths
  1. PACT outperforms the compared baselines across four mathematical benchmarks, demonstrating significant average improvements.
  2. PACT without Importance Sampling (IS), while maintaining other training settings, demonstrates the independent contribution of critic-side IS to mathematical reasoning.
  3. The comparison between BCE and MSE utilizes identical initialization and rollout data, effectively controlling the experimental conditions for critic pre-training.
Critical weaknesses
  1. Appendix B.7 only proves that current-token IS corresponds to a hybrid policy; the experiments do not directly verify whether it reduces the value estimation error relative to the updated policy.
  2. In Appendix D.4, only PACT employs actor-side TIS; the absence of baselines using the same correction makes it difficult to attribute the full performance gains observed in the main results solely to critic training.
  3. Figure 1 does not explicitly utilize held-out rollouts and lacks an ablation study replacing BCE during full RL training; thus, the evidence is insufficient to confirm BCE's contribution to final performance.
  4. The proof in Appendix B.6 does not account for the ratio masking actually employed, and the conditional target mean after masking does not necessarily fall within the [0, 1] range; the conclusion regarding critic consistency cannot be directly applied to the actual algorithm.
Questions for authors

NaN

Reviewer_Yr44OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies token-level credit assignment in long-horizon LLM reinforcement learning with terminal outcome rewards R∈[0,1]R \in [0, 1]. First, over trajectories Y=(q,T1,O1,…,Tτ,Oτ)Y = (q, T _ 1, O _ 1, \dots, T _ \tau, O _ \tau) with filtration Fi=σ(q,T1,O1,…,Ti,Oi)\mathcal{F} _ i = \sigma(q, T _ 1, O _ 1, \dots, T _ i, O _ i), Section 2 introduces three regularity conditions—Completeness (∑i=1τCi=R−E[R∣F0]\sum _ {i=1}^\tau C _ i = R - \mathbb{E}[R \mid \mathcal{F} _ 0]), Prefix Consistency, and Neutrality (E[Ci∣Fi−1]=0\mathbb{E}[C _ i \mid \mathcal{F} _ {i-1}] = 0)—and proves in Theorem 1 that they uniquely determine token credit as the martingale difference Ci=Vi−Vi−1=E[R∣Fi]−E[R∣Fi−1]C _ i = V _ i - V _ {i-1} = \mathbb{E}[R \mid \mathcal{F} _ i] - \mathbb{E}[R \mid \mathcal{F} _ {i-1}]. Section 3 uses this representation to interpret On-Policy Distillation with an ideal QπQ^\pi-tilted teacher (Definition 1, Theorem 2), the gradient unbiasedness of RLOO (Theorem 3), and a variance bound E[∑i=1τCi2∣F0]=Var(R∣F0)≤1/4\mathbb{E}[\sum _ {i=1}^\tau C _ i^2 \mid \mathcal{F} _ 0] = \mathrm{Var}(R \mid \mathcal{F} _ 0) \le 1/4 alongside a GAE error decomposition motivating λ=1\lambda = 1 (Theorem 4).

Second, Section 4 proposes Policy Aligned Critic Training (PACT) (Algorithm 1 in Appendix C, p. 25): in each round, the actor is updated before the critic using λ=1\lambda = 1 advantages (A^t=R−V^t−1\widehat{A} _ t = R - \widehat{V} _ {t-1}), an extra forward pass of the updated actor πθk+1\pi _ {\theta _ {k+1}} is run on the old trajectories Dk∼πθk\mathcal{D} _ k \sim \pi _ {\theta _ k} to compute current-token probability ratios ρt=πθk+1(Tt∣Ft−1)/πθk(Tt∣Ft−1)\rho _ t = \pi _ {\theta _ {k+1}}(T _ t \mid \mathcal{F} _ {t-1})/\pi _ {\theta _ k}(T _ t \mid \mathcal{F} _ {t-1}), and a sigmoid critic V^ϕ,i=σ(zϕ,i)\widehat{V} _ {\phi,i} = \sigma(z _ {\phi,i}) is trained with Binary Cross-Entropy (BCE) against targets Yt=ρtRY _ t = \rho _ t R masked to ρt∈[ρmin⁡,ρmax⁡]=[0,6]\rho _ t \in [\rho _ {\min}, \rho _ {\max}] = [0, 6]. Experiments on Qwen3.5-4B across four math benchmarks (AIME 2025, AIME 2026, BeyondAIME, HMMT Nov. 2025, Table 1) and Qwen3.6-35B-A3B on SWE-bench Verified (Table 2) report 72.87% average Avg@16 and 67.4% Pass@1.

Critical strengths

1. Clean martingale-difference synthesis of token-level credit assignment in Sections 2 and 3.
Framing token credit via Completeness, Prefix Consistency, and Neutrality (Theorem 1), illustrating the necessity of each axiom with explicit counterexamples (Proposition 1 in Appendix B.2.3), and showing how ideal-teacher OPD (Theorem 2), RLOO (Theorem 3), and λ=1\lambda = 1 GAE (Theorem 4) relate to the martingale difference Ci=Vi−Vi−1C _ i = V _ i - V _ {i-1} provides a readable unifying lens for LLM reinforcement learning.

Critical weaknesses

1. The PACT critic objective is theoretically unaligned with the updated policy (Proposition 2) and ill-posed at the sample level when ρtR>1\rho _ t R > 1.
The central algorithmic claim of the paper is that PACT synchronizes the critic with the post-update actor πk+1\pi _ {k+1}. Yet as shown in Appendix B.7 (Proposition 2), replacing the continuation product Wt=∏k=tτρkW _ t = \prod _ {k=t}^\tau \rho _ k with the single-token ratio ρt\rho _ t estimates the value of a one-step hybrid policy π▹tμ\pi \triangleright _ t \mu whose OL(ε)O _ L(\varepsilon) gap from Vt−1πV _ {t-1}^\pi omits all (L−t)(L - t) future token probability shifts Eμ[R∑k=t+1τ(ρk−1)∣Ft−1]\mathbb{E} _ \mu[R\sum _ {k=t+1}^\tau (\rho _ k - 1) \mid \mathcal{F} _ {t-1}]. Over 64k–128k token agentic trajectories, a 1-token correction does not even guarantee moving closer to Vt−1πV _ {t-1}^\pi than the uncorrected target RR. Worse, plugging Yt=ρtR∈(1,6]Y _ t = \rho _ t R \in (1, 6] directly into binary cross-entropy creates a per-sample loss log⁡(1+ez)−Ytz\log(1 + e^z) - Y _ t z that diverges to −∞-\infty as z→+∞z \to +\infty on unique prefixes. This is a foundational soundness flaw in the core method.

2. Severe experimental confounds in the codebase (8$\times$ unique prompts and 2$\times$ training GPUs for PACT vs. GRPO), missing component ablations, and code–paper mismatches.
Inspecting the supplementary code archive reveals multiple issues that invalidate a direct causal attribution of the reported gains in Tables 1 and 2:

  • Uncontrolled prompt diversity and GPU budget in the training launcher (_train_opencode_dapo_qwen3.5_4B_async.sh, lines 28–61): While Section 5.1 states that all methods generate 512 trajectories per round, the launcher configures grpo with 64 prompts × 8 samples on 2 training nodes (16 training GPUs + 16 rollout GPUs), whereas bce_is (PACT) is configured with 512 prompts × 1 sample on 4 training nodes (32 training GPUs + 16 rollout GPUs) plus an extra full actor forward pass after every actor update (actor.py lines 815–876). Thus, PACT sees 8$\times$ more unique training prompts per round and uses 2$\times$ more training GPUs than GRPO.
  • No end-to-end RL ablation separating BCE vs. MSE or Actor-then-Critic update order: PACT simultaneously changes the critic loss (BCE vs. MSE), the update schedule (Actor-then-Critic vs. simultaneous/Critic-then-Actor), the actor objective (DIS [0.7, 6.0] on math), and the critic target (ρtR\rho _ t R). Figure 1 only compares BCE vs. MSE during 160 steps of offline fixed-policy pretraining; there is zero end-to-end RL comparison between PACT (BCE) and PACT (MSE) or PPO + BCE. On SWE-bench Verified (Table 2), even PACT w/o IS is omitted.
  • Code–paper inconsistencies and single-seed reporting: Beyond the masked-loss denominator contradiction between Algorithm 1 (∑m\sum m) and critic_bce_loss.py (total action tokens), Appendix D.4 (p. 26) states that PACT uses actor TIS in [0, 2] and Table 3 specifies math actor DIS [0.7, 6.0], whereas scripts/OpenCodeDAPO/Qwen3.5-4B/README.md (lines 38–40) sets BCE_ACTOR_USE_TIS=false and BCE_ACTOR_EPS_LOW=0.4, BCE_ACTOR_EPS_HIGH=2.0 ([0.6, 3.0]). All results in Tables 1–2 are single-seed point estimates without error bars, and closely related value-based baselines (VC-PPO, VAPO, RLOO) are absent.
Questions for authors
  1. Assessment-critical (Theoretical validity of single-token IS in Proposition 2):
    Given that Appendix B.7 Proposition 2 shows V~t−1π,μ=Eμ[ρtR∣Ft−1]=Vt−1π+OL(ε)\widetilde{V} _ {t-1}^{\pi,\mu} = \mathbb{E} _ \mu[\rho _ t R \mid \mathcal{F} _ {t-1}] = V _ {t-1}^\pi + O _ L(\varepsilon) omits the entire continuation sum Eμ[R∑k=t+1τ(ρk−1)∣Ft−1]\mathbb{E} _ \mu[R\sum _ {k=t+1}^\tau (\rho _ k - 1) \mid \mathcal{F} _ {t-1}], under what formal conditions is ∣V~t−1π,μ−Vt−1π∣<∣Vt−1μ−Vt−1π∣\lvert \widetilde{V} _ {t-1}^{\pi,\mu} - V _ {t-1}^\pi \rvert < \lvert V _ {t-1}^\mu - V _ {t-1}^\pi \rvert when τ−t∼104\tau - t \sim 10^4 tokens? How does PACT perform if you use a kk-step continuation product ∏j=tmin⁡(τ,t+k−1)ρj\prod _ {j=t}^{\min(\tau, t+k-1)} \rho _ j or the first-order estimator YtFO=R(1+∑k=tτ(ρk−1))Y _ t^{\mathrm{FO}} = R(1 + \sum _ {k=t}^\tau (\rho _ k - 1)) derived in Appendix B.7?

  2. Assessment-critical (Unbounded-below per-sample BCE loss for ρtR>1\rho _ t R > 1 and end-to-end BCE vs. MSE ablation):
    In your main runs with [ρmin⁡,ρmax⁡]=[0,6][\rho _ {\min}, \rho _ {\max}] = [0, 6], what fraction of positive-reward tokens have Yt=ρtR>1Y _ t = \rho _ t R > 1, and what fraction of per-token BCE losses are negative? How does end-to-end RL performance in Table 1 change if you (a) replace BCE with MSE inside PACT (PACT-MSE), (b) clamp targets to [0,1][0, 1] (min⁡(ρtR,1)\min(\rho _ t R, 1)), or (c) apply ρt\rho _ t as an outer sample weight ρt⋅BCE(σ(z),R)\rho _ t \cdot \mathrm{BCE}(\sigma(z), R) so that the per-sample loss remains bounded below by 0?

  3. Assessment-critical (8$\times$ unique-prompt confound and code–paper discrepancies):

    • In Table 1, did PACT use 512 prompts × 1 sample per round on 32 training GPUs while GRPO used 64 prompts × 8 samples on 16 training GPUs, as configured in _train_opencode_dapo_qwen3.5_4B_async.sh? What are the results when GRPO, PPO (λ=1.0\lambda = 1.0), PACT w/o IS, and PACT are run under identical unique-prompt counts per round and identical GPU/FLOP budgets across ≥3\ge 3 random seeds?
    • Which masked-loss normalization produced Tables 1–2: dividing by accepted tokens ∑m\sum m (Algorithm 1) or dividing by total action tokens (critic_bce_loss.py line 461)? And was BCE_ACTOR_USE_TIS set to true ([0, 2], Appendix D.4) or false (README.md)?
Reviewer_ZN8kOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that despite credit assignment, the process of assigning how final outcome (scalar reward) attributes to independent actions, is important for RL, existing work lack mathematical characterization. The authors argue that common formulations like treating AR generation as the markov decision process fail to do so. In this work, they formulate three regularity conditions and derive the unique credit-assignment formulation satisfying them. They use this formulation to motivate the design of PACT, a practical actor–critic training procedure with importance-corrected critic targets and a BCE critic loss. They present empirical results to support the resulting training design.

Critical strengths
  1. The critic-IS ablation provides positive evidence for a useful practical improvement under good controlled experiments. This supports an incremental contribution even if the theoretical framing adds little.
Critical weaknesses
  1. The theoretical contribution is not convincingly established beyond standard TD/advantage reasoning and martingale identities. Recasting familiar properties as axioms does not itself establish a new credit-assignment principle, and I do not see what substantive new insight this characterization adds to the training design. This weakens a major claimed contribution.

  2. The PPO-PACT comparison seems unfairly favorable to PACT. Changes to the critic loss and actor-side corrections can already improve performance, and these known improvements should be available to the relevant baselines too. The full gain over PPO should not be attributed to the proposed critic correction. The no-IS ablation provides more relevant evidence for its incremental benefit.

Questions for authors

What substantive result goes beyond standard TD/advantage reasoning and martingale properties? In particular, what new insight from the proposed axioms justifies the specific training-objective design?

Why are the known critic-loss improvements and actor-side corrections not applied to the relevant baselines? Are comparisons available with these choices matched across methods? I acknowledge that PACT w/o IS provides useful evidence for the incremental critic-IS benefit. My question concerns how much the comparisons establish beyond combining these other improvements.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.