acceptodds
Under review as a conference paper at ICLR 2027

Individual Credit, Joint Effects: Understanding Credit Assignment in Agentic RL

Abstract

Credit-assignment methods give each action of an agent its own learning signal, and most recent work improves how these signals are constructed. How the signals change the policy once they enter training, which should inform their design, remains poorly understood. In GiGPO training on a controlled agentic task, 37.5% of actions with nonzero credit change probability against it. To trace why, we decompose each Adam update into contributions from individual current and past actions, which we call sources. After the first update, removing the four most strongly opposing sources restores the credit direction in 75.8% of reversals with count- and norm-matched controls, against 16.7% for other opposing sources. From the third update on, most reversals also require removing historical sources, partly because past gradients gain weight in Adam's first moment. Compared with candidate actions, the removed sources more often perform the same kind of operation as the target when their credit is opposite, and share one of its immediate effects when their credit has the same sign. A credit signal may thus reach related actions in these two ways, within an update and, through momentum, in later ones, which raises the question of how much history the optimizer should retain. We also test a history-retention adjustment that uses only existing optimizer state: with GiGPO and GRPO on ALFWorld and WebShop, uniformly attenuating retained momentum has no consistent effect on final success, consistent with attenuation scaling both helpful and harmful history equally.

Primary area:
reinforcement learning
Keywords:
credit assignment, reinforcement learning, language model agents, policy optimization, process supervision, empirical analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_PRpfOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper examines why actions sometimes move against the credit assigned to them under GiGPO. To do this, it decomposes gradients per action to assess which sources contribute the most to credit reversal, then checks if the credit recovers when those sources are removed. It finds that credit recovers quickly in most cases, and that frequent causes of reversal are other actions with the same operation (in cases where source and target credit signs oppose) and other actions with the same effect (in cases where source and target credit signs agree). The paper finally examines whether credit reversal can be reduced by modifying the optimizer to reduce weight placed on previous updates, and finds that this has no clear effect.

Critical strengths

This paper clearly motivates its problem, supports its claims well, and is written clearly. Aside from a few small clarifications, I have no concerns about this paper.

Critical weaknesses

None

Questions for authors
  1. Under the controls in Section 3.2, why is k=2 not compared?
  2. Why does Section 3.3 only use 19 seeds rather than the 21 seeds used in the previous experiment?
1 comment

Dear Reviewer PRpf,

We thank you for recognizing the motivation, presentation, and evidence of the paper, and for two precise questions. We respond to each below.

Q1: Why is k=2k=2 not compared under the controls in Section 3.2?

"Under the controls in Section 3.2, why is k=2 not compared?"

The cumulative sweep covers k∈{1,2,4,8,16,32}k \in \{1, 2, 4, 8, 16, 32\} (Table 16), and k=2k=2 restores 63.4% of reversals, between 47.3% at k=1k=1 and 80.6% at k=4k=4. The matched controls are costly: for each target, kk, and arm, we search up to 20,000 norm-matched subsets, which took 93,478 forward evaluations for three cutoffs. We therefore chose k=1k=1 (a single source), k=4k=4 (several sources), and k=8k=8 (above 90% recovery). The ordering top opposing > other opposing > supporting holds at every update and every seed for all three cutoffs (Tables 21–22), so adding k=2k=2 would not change the comparison. Appendix D.2 now states this reason.

Q2: Why does Section 3.3 use 19 seeds instead of 21?

"Why does Section 3.3 only use 19 seeds rather than the 21 seeds used in the previous experiment?"

The 19 seeds are the intersection of two audited sets (Appendix E.1). Seed 10 ran update 1 only, so 21 seeds have all five updates. We froze the update-1 audit protocol on seeds 1–20 before running seeds 21–22, so the prompt was not tuned on later data. Table 1 pools updates 1–5 and therefore uses the seeds audited at every update: 1–9 and 11–20. The update-1 rates over all 20 seeds (96.4% and 66.6%; Table 26) match the 19-seed entries in Table 1 (97.2% and 66.5%), so the restriction does not bias the result. The Table 1 caption and Appendix E.1 now say this.

Comments from the other reviewers led to a rewritten Abstract, a new Figure 1 built from the controlled results, added baselines (Table 13) and deletion controls (Table 19), and a new Section 4; the earlier capping tables are in Appendix F.

If time permits, we would be grateful if you could also look at the revised PDF and share any further comments.

Reviewer_YzsZOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents an empirical study of the credit assignments in GiGPO training with detailed analysis of a number of questions for the purpose of deepening our understanding of the behavior of credit assignments. However, the paper is very hard to follow and not written clearly to articulate a research question in the context of the previous work or a clear hypothesis that would be tested in the experiments. Based on the paper, the contributions are the following (copied from the paper): (1) a record-level expansion of Adam updates in GiGPO training into peraction sources, including those carried by momentum; (2) matched deletion controls and blinded audits that locate current and historical sources of probability reversals and identify two recurring relations between sources and the actions they affect; and (3) a test of history retention computed without per-action gradients, with GiGPO and GRPO on ALFWorld and WebShop, showing no consistent benefit from uniform attenuation.

Critical strengths
  • Credit assignments are important in reinforcement learning. The paper presented some empirical findings about credit assignments in GiGPO which may be useful for improving credit assignments in GiGPO.
Critical weaknesses
  • The main weakness is that the paper is poorly written with too many technical details that are hard to follow. There is a lack of high level story of research, which should include a clearly defined research problem, a discussion of existing work that attempted to address the research problem, their limitations, how the proposed work would address those limitations, etc...
  • The flow of the paper is incoherent with a lof of low-level details scattered here and there; even in the abstract, numerical values were mentioned without a context.
  • The paper reads more like notes of research concatenated together.
Questions for authors
  • Can you tell your research story at a high-level to clearly articulate what's exactly your research question? Is it new? What previous work has attempted to address the question or a similar question?
1 comment

Dear Reviewer YzsZ,

We thank you for recognizing the importance of credit assignment in agentic RL and for the candid comments on readability. We respond to each point below.

W1: Flow of the paper and numbers without context

"The flow of the paper is incoherent with a lof of low-level details scattered here and there; even in the abstract, numerical values were mentioned without a context."

We agree. The earlier draft listed results before saying what tied them together. The Abstract is now one paragraph that follows a single question and contains no raw percentages. Figure 1 now shows the controlled results in four panels, so the reader sees the evidence first. Numbers in Section 2 now come with reference points: updating one action alone reverses 0%, tiny SGD steps that do not learn reverse 46.0%, and training settles near 37% (Figure 1a, Table 13). Exploratory results and per-seed tables moved to the appendix.

Q1: Research question, novelty, and prior work

"Can you tell your research story at a high-level to clearly articulate what's exactly your research question? Is it new? What previous work has attempted to address the question or a similar question?"

Our question is: once scalar credits enter batched Adam updates of a shared-parameter agent, how do they move action probabilities? Prior work answers neighboring questions (Section 5). Credit-assignment methods (GiGPO, VinePPO, AgentPRM, Auzina et al.) build better weights before training and judge them by final return. Learning-dynamics studies (Ren & Sutherland, Cheng et al., Liu et al.) examine coupling within a single step. Gradient surgery (PCGrad, CAGrad) resolves conflicts between separate task losses, which single-objective RL does not have. None follows credits through multi-turn updates with optimizer momentum.

Our answer has three parts. Updating one action alone never reverses it, yet about 37% of actions reverse in batched steps, so other actions often outweigh a target's own credit (Section 2). Holding Adam's second-moment scaling fixed splits each step into current and past terms, and deletion shows that a few of them cause each reversal, often from past batches (Section 3.2). Audits and decision-slot checks show why: actions doing the same operation share tokens, and alternatives with the same effect compete in one softmax (Section 3.3). Credits score actions by task value, but updates couple them through these shared structures. A four-seed WebShop proof-of-concept then tests one consequence of this view: rescaling coarse reward terms can steady noisy online training (Section 4, Table 3).

Your comments on readability shaped most of this revision. We would be very grateful if you could read the revised PDF again, especially the Abstract, Section 1, and Figure 1, and we would sincerely welcome any further comments.

Reviewer_BoxiOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how credit assigned to each action affects the policy during rl training and studies this empirically from the perspective of adam optimzier. Specifically, it decomposes Adam updates into contributions from actions in the current and previous batches, then uses deletion experiments to examine why some action probabilities move against their assigned credit. It also tests an adjustment to Adam’s momentum. The matched deletion experiments are useful, but I found the paper very difficult to follow and have several concerns about its contribution and experimental evidence.

  • The paper needs a clearer explanation of what each part of the analysis is meant to show. The paper moves from measuring probability changes to decomposing updates, interpreting relationships between actions, and modifying momentum, without sufficiently explaining how these steps are related and support with their main argument. For example, the statement that “positive credit tends to raise an action’s probability and negative credit tends to lower it” only holds locally and is not generally correct, as parameters are shared, so this point is unclear.

  • The contribution appears primarily empirical. Eq 2 measures the change in an action’s log-prob before and after an update. Previous work, like [1], already studies how training on one example affects the probabilities of other ones. Also, [2] uses the change in log-pob assigned to the correct target across interaction steps as an intermediate reward. Here, Eq. 2 measures the probability change after a parameter update. While i understabd these methods differ in what changes between measurements and how the resulting signal is used, but this connection need to discussed. The paper should explain what is new about their analysis and which findings add to existing work on probability changes, gradient interactions, and optimizer history.

[1] Learning Dynamics of LLM Finetuning. Yi Ren and Danica J. Sutherland.

[2] Intrinsic Credit Assignment for Long Horizon Interaction. Ilze Amanda Auzina, Joschka Strüber, Sergio Hernández-Gutiérrez, Shashwat Goel, Ameya Prabhu, and Matthias Bethge.

  • The main experiments contains only limited setting, i.e. Qwen3.5 0.8B trained with LoRA, and five updates per complete run, but it is not clear whether the findings hold for other models or later stages of training. This matters because some of the reported behavior depends on initialization and the increasing contribution of past gradients during the first few updates. The ALFWorld and WebShop experiments test momentum attenuation, and they do not repeat the detailed attribution analysis in those environments. Also, the experiments are not done in a carefully controlled setting. For example, the momentum experiment changes several things at once such as attenuating history reduces supporting and opposing contributions, and it also changes the update magnitude. Hence, it is difficult to attribute the results to the effect of optimizer history.

  • It is unclear when the observed probability reversals are harmful. Optimizing a model with shared parameters involves tradeoffs between examples, so an action moving against its assigned credit does not necessarily imply a problem with learning. Indeed, the controlled runs improve substantially despite frequent reversals. Although the authors acknowledge this , the paper does not show when reducing reversals would improve task performance. The deletion experiments also keep the observed Adam scaling fixed. They explain contributions to a particular update, but do not establish what would happen if the corresponding experiences were removed from training.

In summary, since this paper is primarily an empirical paper, it requires comprehensive experiments and controlled one. However, in its current form, the paper lacks those and only covers a very limited setting.

Critical strengths

see above

Critical weaknesses

see above

Questions for authors

see above

1 comment

Dear Reviewer Boxi,

We thank you for finding the matched deletion experiments useful and for the detailed critique of structure, positioning, and controls. We respond to each point below.

W1: How the analysis stages connect, and the credit-sign phrasing

"The paper moves from measuring probability changes to decomposing updates, interpreting relationships between actions, and modifying momentum, without sufficiently explaining how these steps are related..."

We agree and have rewritten Sections 1–3 around one question: once scalar credits enter batched Adam updates, how do they move action probabilities? You are right that "positive credit raises probability" holds only locally. What holds exactly is narrower: updating one action alone never reverses it (0 of 4,077; Table 13). A reversal in a batched step is therefore a trace of other actions acting through shared parameters, and each later stage asks one follow-up question: which terms cause it (Sections 3.1–3.2), how they relate to the target (Section 3.3), and whether the pattern persists at full scale (Section 4).

W2: What is new relative to [1] and [2]

"The paper should explain what is new about their analysis and which findings add to existing work on probability changes, gradient interactions, and optimizer history."

We now discuss both directly in Section 5. Auzina et al. [2] compare a target answer's log-probability across interaction turns with parameters fixed, to build a reward before any update. We take credits as given and measure how one actual update moves the sampled actions. Relative to [1] and to single-turn work (Cheng et al.), which study coupling within one step, three findings are new. First, under Adam the opposing terms also come from past batches: across updates 2–5, 53.7% of minimal recovering sets include a historical record, and 20.1% recover by removing history alone (Table 23). Second, the pull is sparse and directional: in updates 2–5, removing the top 4 opposing terms restores 75.8% of targets, against 16.7% and 0.6% for norm-matched controls (Figure 1b), and removing the largest-norm terms gives zero median change (Table 19). Third, the pull follows task structure: opposite-credit pairs share an operation, same-credit pairs share an immediate effect, and at shared decision slots the sign follows token choice (1,357 of 1,357 same-token pairs positive, 623 of 628 different-token pairs negative; Tables 1–2).

W3: Limited setting and the confounded momentum experiment

"The main experiments contains only limited setting... the momentum experiment changes several things at once..."

We agree on both. On the setting, reversals are not a 5-step or Adam artifact: step-norm-matched plain SGD gives 38.2%, Adam without momentum 36.6%, and 15-update runs keep 36.5% at 99.9% test success (Table 13). In regularized WebShop GRPO runs, 32.3% (50-step Qwen2.5-1.5B LoRA) and 22.7% (70-step full-parameter Qwen3-4B) of nonzero-credit actions still reverse per step, and momentum supplies about two-thirds of each reward step (Appendix G.3). On the confound, your reading was right: uniform capping shrank step size and KL together (Appendix F.2), so it now sits in Appendix F. The new Section 4 keeps the AdamW moments and KL untouched, rescales only coarse reward terms by their sign agreement with a verified reference direction, and compares against a random-sign arm with matched rates. With four paired seeds we treat it as a proof-of-concept: random-sign mainly prevents one collapsing control seed (2/4 wins), while rescaling the current batch leads control on 4/4 seeds and random-sign on 3/4 for late and trajectory-wide scores (Table 3, Figure 2b).

W4: When reversals matter, and what fixed-DtD_t deletion shows

"It is unclear when the observed probability reversals are harmful... They explain contributions to a particular update, but do not establish what would happen if the corresponding experiences were removed from training."

We agree with both distinctions and state them in Sections 3.2 and 6. Individual reversals do not slow learning in our runs: after controlling for update and solved count, the slope on newly solved tasks spans zero (-6.0 tasks per +10 points, 95% CI [-13.7, 2.2]; Appendix B.4). What can hurt is the combined direction drifting under noise in small-batch online RL, as in the collapsing WebShop control seed, which is why Section 4 adjusts coarse reward terms instead of individual reversals. Fixed-DtD_t deletion attributes one realized step and does not predict retraining without a sample. We now call its outcome sign restoration and report update 1 separately from updates 2–5 (Section 3.2).

We have reworked the paper substantially in response to your comments, especially its structure and controls. We would be very grateful if you could take another look at the revised PDF, and we would sincerely welcome any further comments.

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We thank Reviewer PRpf for finding the problem well motivated and supported, Reviewer YzsZ for recognizing the value of understanding credit assignment in agentic RL, and Reviewer Boxi for finding the matched deletion experiments useful. The reviews share three concerns: the paper read as notes without one story, several numbers lacked a reference point, and Section 4 changed several things at once. All Section 2–3 runs and measurements are unchanged, and the revised PDF addresses these concerns as follows.

Story. The Abstract is now one paragraph that follows a single question: once scalar credits enter batched Adam updates, how do they move action probabilities? Figure 1 now shows the controlled results directly in four panels. Section 3.4 states what the analysis adds beyond the loss in Eq. (1): credits score actions by task value, but updates couple actions through shared operation tokens and softmax competition across current and momentum batches.

Reference points. Updating one action alone reverses 0 of 4,077 actions, and tiny SGD steps that do not learn reverse 46.0% (Table 13). During training the rate stays near 37% while test success rises to 93.9% (Figure 1a). Deletion results are now reported separately for update 1, where Adam behaves like sign descent, and for updates 2–5, where removing the top 4 opposing terms restores 75.8% of reversed targets against 16.7% and 0.6% for norm-matched controls (Figure 1b, Appendix D.3).

Section 4. The earlier momentum-capping test shrank step size and KL together, as Reviewer Boxi pointed out, and now sits in Appendix F. The new Section 4 is a four-seed proof-of-concept on full-parameter Qwen3-4B WebShop. It keeps KL and the AdamW moments untouched, rescales only coarse reward terms, and compares against a random-sign baseline with matched rates. We report it as directional evidence: random-sign mainly prevents one collapsing seed (2/4 wins over control), while rescaling the current batch with a verified reference direction leads control on 4/4 seeds for late and trajectory-wide scores (Table 3, Figure 2b).

We revised the paper carefully around these comments. We would be very grateful if the reviewers could read the revised PDF again and share any further comments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.