Let Credit Follow Computation: Architecture-Aware Credit Transport for Large Language Model Reinforcement Learning

Large-language-model reinforcement learning has substantially improved reward modeling, rollout sampling, baselines, and policy-update geometry, but it usually leaves one component architecture-agnostic: the operator that transports delayed evidence into token-level credit. Fixed-discount generalized advantage estimation (GAE) applies the same geometric kernel to every trajectory, while group-relative methods broadcast a response-level statistic to all tokens. We identify this discrepancy as an architecture–credit mismatch and introduce computation-conditioned credit transport (CCT), in which a detached statistic of the behavior policy's internal computation parameterizes a trajectory-specific causal kernel. Its concrete instantiation, CompPO, maps native attention concentration to a bounded retention gate, uses the gate in both the one-step bootstrap and a path-dependent advantage trace (Comp-GAE), and co-designs a $14.4$M transport-aligned critic (TAC) that reuses actor hidden states and the policy KV cache. The external reward and clipped PPO objective remain unchanged, and a constant gate recovers fixed-coefficient GAE. Across five Qwen3-4B seeds, CompPO reaches $61.4\%$ final held-out accuracy (95% CI $[60.8,62.0]$), versus $53.8\%$ $[52.9,54.7]$ for GRPO from the same $3\times 3$ screen, under the same $4\times 4$ trajectory budget, with a best-to-final gap of $0.3$ versus $2.4$ points. Under that shared budget, CompPO reaches GRPO's $56.4\%$ peak at step $33$. A controlled gate-by-critic factorial yields a $+2.4$-point final interaction $[1.9,2.9]$; shuffling and position-only controls show that trajectory-specific alignment contributes beyond mean scale and position. CompPO is stable in $10/12$ runs in a matched PPO stress grid versus $3/12$ for PPO, and improves frozen greedy macro accuracy over GRPO by $4.2$ and $3.8$ points on Qwen3-4B and Llama-3.1-8B-Instruct. These results establish policy-internal computation as a useful variable for RL credit estimation and motivate architecture-aware Bellman operators, traces, and critics.