Same Observation, Different Futures: Credit Assignment via Dynamics Modeling

LLM agents are moving from single-turn answers toward long-horizon workflows spanning dozens of decisions and tool calls. As the interaction horizon grows, a sparse terminal reward must be attributed across increasingly many decisions, progressively weakening the supervision available for each individual decision. Credit assignment therefore becomes a key bottleneck in agentic RL. A common approach is to estimate intermediate history values and use them to construct step-level advantages. Existing methods group histories using heuristic local matching and estimate values from within-group Monte Carlo (MC) returns. This can introduce bias by grouping histories with different continuation dynamics and increase variance by ignoring trajectories outside the group. In this work, we propose Credit Assignment via global DYnamics (CADY), a structured credit assignment method that models global trajectory dynamics. CADY maps histories with similar continuation dynamics to similar posterior distributions over latent states, mitigating the bias induced by heuristic grouping. It further exploits the global coupling among latent states for structured value estimation. This allows information from all rollouts to be more effectively leveraged through the shared transition dynamics. We further provide a theoretical analysis explaining why global dynamics-based value estimation can achieve lower first-order asymptotic variance than a simple MC estimator. Experiments on three challenging long-horizon agentic benchmarks show that CADY produces more accurate history-value estimates and substantially higher task success rates. Even in a tool-use environment with roughly 500 API operations, CADY can still separate locally indistinguishable histories with divergent continuations, demonstrating its scaling potential.