acceptodds
Under review as a conference paper at ICLR 2027

Let Credit Follow Computation: Architecture-Aware Credit Transport for Large Language Model Reinforcement Learning

Abstract

Large-language-model reinforcement learning has substantially improved reward modeling, rollout sampling, baselines, and policy-update geometry, but it usually leaves one component architecture-agnostic: the operator that transports delayed evidence into token-level credit. Fixed-discount generalized advantage estimation (GAE) applies the same geometric kernel to every trajectory, while group-relative methods broadcast a response-level statistic to all tokens. We identify this discrepancy as an architecture–credit mismatch and introduce computation-conditioned credit transport (CCT), in which a detached statistic of the behavior policy's internal computation parameterizes a trajectory-specific causal kernel. Its concrete instantiation, CompPO, maps native attention concentration to a bounded retention gate, uses the gate in both the one-step bootstrap and a path-dependent advantage trace (Comp-GAE), and co-designs a M transport-aligned critic (TAC) that reuses actor hidden states and the policy KV cache. The external reward and clipped PPO objective remain unchanged, and a constant gate recovers fixed-coefficient GAE. Across five Qwen3-4B seeds, CompPO reaches final held-out accuracy (95% CI ), versus for GRPO from the same screen, under the same trajectory budget, with a best-to-final gap of versus points. Under that shared budget, CompPO reaches GRPO's peak at step . A controlled gate-by-critic factorial yields a -point final interaction ; shuffling and position-only controls show that trajectory-specific alignment contributes beyond mean scale and position. CompPO is stable in runs in a matched PPO stress grid versus for PPO, and improves frozen greedy macro accuracy over GRPO by and points on Qwen3-4B and Llama-3.1-8B-Instruct. These results establish policy-internal computation as a useful variable for RL credit estimation and motivate architecture-aware Bellman operators, traces, and critics.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, large language models, credit assignment, proximal policy optimization, actor-critic, attention mechanism, policy optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qeouOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

The paper starts from the observation that GRPO gives every token the same credit, while PPO with GAE passes credit back with a fixed discount; neither looks at how the model actually computed each token. The idea is to use attention for this: at each token, the authors measure how concentrated the final-layer attention is, and map it to a gate that replaces the fixed discount in GAE. When attention is concentrated, more of the later reward is passed back to that token; when it is spread out, less. They also add a small critic on top of the actor’s hidden states, which mixes the current token’s state with the attended history using the same gate, so that the critic and the credit transport are aligned. The rest of PPO is unchanged. The authors claim that this outperforms GRPO.

Critical strengths

The idea of using attention to set the discount in GAE is interesting.

Critical weaknesses
  1. The writing is hard to follow. A fairly simple change (replacing the fixed discount in GAE with a per-token value computed from attention) is wrapped in a lot of new terms (architecture–credit mismatch, CCT, CompPO, Comp-GAE, TAC, transport operator, retention gate, ...), and so many sentences describe simple things in an abstract way.

  2. From the shuffle and position-only controls, it is not clear that the gain really comes from attention. Shuffling the gates across tokens (so they no longer match the attention at all) still gives results close to the full method, and both are far above GRPO.

  3. The average gate is around 0.6 per token, while responses go up to 16K tokens. With that much decay, the final reward basically disappears after a few dozen tokens, so it is unclear how this “transports delayed credit” through long reasoning, as the paper claims.

  4. Several claims go beyond what the experiments show. For example, the abstract says the results “establish” internal computation as a useful variable for credit, although the shuffled-gate control performs close to the full method; the discussion says CCT composes with DAPO/GSPO/process rewards, but no such combination is tested; and statements like “learned values faithfully track” the targets (EV is around 0.5) or that the paper “re-evaluates” the move away from critics seem too strong for one 4B model on math with 200 training steps.

Questions for authors

Accuracy on a 500-problem set should be a multiple of 0.2%, which holds for all single-run numbers in the paper, but several seed-level values in Tables 8 and 10 (e.g., 61.5, 51.9) are not. How was ID accuracy computed for the confirmation runs?

Reviewer_NpJqOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies credit assignment using information from the architecture side, i.e., attention. They leveraged the attention pattern to calculate how outcome reward propagate to previous tokens (i.e., discount factor) and proposed a lightweight critic that only has 0.36% parameters. With these designs, the method achieves strong performance compared to GRPO and PPO.

Critical strengths
  1. The method is well-motivated and sound. The paper is clearly written and easy to follow.
  2. Empirical results are strong compared to GRPO and PPO, with .95 CI reported. Analysis and ablations are provided, justifying their design.
  3. The design of TAC achieves a much lower cost compared to a standard LLM RL critic.
Critical weaknesses
  1. My main concern is about the evaluation. How does it compare to better RL methods like GRPO with higher clip_high?
Questions for authors
  1. Is scaling up the parameters of the critic helpful to the training? Say, what if it's 10x larger.
Reviewer_hmtWOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies token-level credit assignment in reinforcement learning for long-chain-of-thought large language models (Section 1–3). It decomposes credit assignment into delayed evidence, a credit transport operator, and outer update geometry (Section 1 Eq. (1)–(2)), arguing that fixed-parameter GAE (Section 3 Eq. (3)–(5)) and group-relative broadcasting (GRPO) ignore the behavior policy's trajectory-specific internal computation (Definition 1). The authors propose Computation-Conditioned Credit Transport (CCT) and its instantiation CompPO (Section 3–4, Appendix A Algorithm 1), which computes a length-normalized Herfindahl attention-concentration statistic ct∈[0,1]c _ t \in [0, 1] from the final transformer layer (Section 4.1 Eq. (11)), maps it to a retention gate κt=κlo+(κhi−κlo)σ(τ(ct−1/2))\kappa _ t = \kappa _ {\mathrm{lo}} + (\kappa _ {\mathrm{hi}} - \kappa _ {\mathrm{lo}}) \sigma(\tau(c _ t - 1/2)) with (κlo,κhi,τ)=(0.1,0.9,4)(\kappa _ {\mathrm{lo}}, \kappa _ {\mathrm{hi}}, \tau) = (0.1, 0.9, 4) (Eq. (12)), and uses κt\kappa _ t simultaneously in the TD residual δtκ\delta _ t^\kappa (Section 3 Eq. (7)), the recursive Comp-GAE trace Atκ,λA _ t^{\kappa, \lambda} (Eq. (8)), and a 14,428,163-parameter Transport-Aligned Critic (TAC) that mixes locally pooled and causal top-KK (K=64K=64) routed history representations (Section 4.2 Eq. (13)–(18), Figure 1). Experiments on DAPO-Math-17K with Qwen3-4B (and frozen-benchmark transfer on Llama-3.1-8B-Instruct) report 5-seed accuracy gains over GRPO on a 500-problem in-distribution (ID) set (61.4% vs. 53.8%, Section 6.1 Table 1, Figure 2), a positive 2×22 \times 2 gate-by-critic interaction (+2.4 points, Section 6.2 Table 2), gate-permutation controls (Section 6.3 Table 3), an untuned PPO stress grid (Section 6.4 Table 4), and point-estimate gains across seven frozen math benchmarks (Section 6.5 Table 5).

Critical strengths
  1. Clear conceptual decomposition of credit assignment and unified gate usage. Separating delayed reward evidence, temporal transport, and outer policy-update geometry (Section 1 Eq. (1)–(2)) provides a clean conceptual lens, and feeding the same trajectory-conditioned retention gate into the TD bootstrap (Eq. (7)), eligibility trace (Eq. (8)), and critic value mixture (Eq. (18)) is more coherent than post-hoc loss reweighting.

  2. Informative 2×22 \times 2 factorial and gate-randomization experimental design. The seed-matched 2×22 \times 2 gate-by-critic factorial design (Section 5 Eq. (20), Section 6.2 Table 2) and the fixed-mean, global-shuffle, and position-only controls (Section 6.3 Table 3) are well-structured diagnostic tests for probing whether trajectory-specific gating and critic architecture interact positively.

Critical weaknesses
  1. The primary evaluation benchmark is part of the online training control loop and hyperparameter selection. Using the same 500-problem ID set for 3×33 \times 3 hyperparameter selection (Table 1B), every-5-step closed-loop phase-controller feedback (Appendix C.1 Tables 16–18), and the headline step-200 evaluation (Table 1A, Tables 2–4) invalidates the primary accuracy numbers as independent generalization estimates. Meanwhile, the frozen benchmark evaluations in Section 6.5 Table 5 report only single-checkpoint point estimates without multi-seed variance, and the odd-tenth percentages (52.9%, 61.5%, etc.) on the 500-problem ID set in Tables 8 and 10 contradict the 0.2% granularity of a 500-item exact-match test.

  2. Severe code-versus-paper mismatch, missing strong baselines, incomplete model scaling, and purely analytic systems numbers.

    • Irreproducible artifact: As detailed under Soundness, the supplementary ZIP (attn-nm-rl) differs from the manuscript across response length (1024 vs. 16K), parameter update mode (LoRA vs. full parameter), advantage estimator (use_nm_gae=False with logaddexp dynamic returns vs. Eq. (8)), reward/loss terms (use_value_aware_reward, use_attn_mi, aw_hdc), critic architecture and parameter count, absence of the phase controller, and dense T2T^2 attention allocation.
    • Missing baselines and asymmetric model matrix: Despite training on DAPO-Math-17K and surveying fine-grained credit and group-relative methods in Section 2 and Table 6 (DAPO, Dr. GRPO, GSPO, GRPO-λ, OAR, OPPO, S-trace, VinePPO, AdaGamma), the main experiments compare only against basic GRPO and an untuned 150-step PPO stress grid without the controller (Table 4). Within the Qwen3 family only 4B is tested (skipping 1.7B, 8B, 14B), while Llama-3.1-8B-Instruct appears only as a single-run point estimate in Table 5 without training curves or ablations. Finally, all memory and compute numbers in Tables 19–22 are analytic formulas rather than measured wall-clock runtime, throughput, or peak HBM.
Questions for authors
  1. Reconciling the Supplementary ZIP with the manuscript: Why does the submitted supplementary archive (attn-nm-rl) reference a non-existent train.py, configure 1024-token LoRA training with use_nm_gae=False, use_value_aware_reward=true, use_attn_mi=true, and aw_hdc in qwen3_4b_dapo17k.yaml, define three critic classes (ValueNetwork v1/v2/v3) none of which matches Eq. (13)–(18) or 14,428,163 parameters, stub out the adaptive phase controller in dp_actor.py / dp_critic.py, and materialize dense buf_rlen x buf_rlen attention tensors in qwen3_nmdp_patch.py? Can you provide the exact code, configurations, and raw logs that produced Tables 1–5, along with the exact layer/projection dimensions of TAC?

  2. Eliminating closed-loop ID-set contamination and explaining odd-tenth accuracies on 500 problems:

    • How can the 500-problem ID set be treated as a held-out evaluation benchmark when it drives online phase-controller hyperparameter changes every 5 steps (Appendix C.1 Tables 16–18) and 3×33 \times 3 grid selection? Can you report 5-seed results on a strictly untouched test split?
    • On a 500-problem exact-match benchmark, why do Seeds 17, 123, 256, and 2026 in Tables 8 and 10 contain 13 odd-tenth percentages (such as 52.9% = 264.5/500 and 61.5% = 307.5/500), whereas Seed 42 and all 36 entries in Tables 1B, 9, and 15 are exact multiples of 0.2%? Also, how do you reconcile Table 7 (Selected KL = 1e-3 for GRPO) with Table 16 (Stable phase KL = 0.0020)?
  3. Kernel attenuation over 16K rollouts and comparisons to strong baselines: With κt≤0.8046\kappa _ t \le 0.8046 (Eq. (12)) and λ=0.95\lambda = 0.95, the direct trace weight Kt,uκ,λK _ {t,u}^{\kappa, \lambda} falls to ≤1.46×10−6\le 1.46 \times 10^{-6} over 50 tokens. In 16K-token rollouts with terminal binary rewards, how much non-zero credit reaches early and middle tokens, and how does CompPO compare against DAPO, Dr. GRPO, or GRPO-λ under measured wall-clock time and peak HBM?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.