This paper studies how credit assigned to each action affects the policy during rl training and studies this empirically from the perspective of adam optimzier. Specifically, it decomposes Adam updates into contributions from actions in the current and previous batches, then uses deletion experiments to examine why some action probabilities move against their assigned credit. It also tests an adjustment to Adam’s momentum. The matched deletion experiments are useful, but I found the paper very difficult to follow and have several concerns about its contribution and experimental evidence.
-
The paper needs a clearer explanation of what each part of the analysis is meant to show. The paper moves from measuring probability changes to decomposing updates, interpreting relationships between actions, and modifying momentum, without sufficiently explaining how these steps are related and support with their main argument. For example, the statement that “positive credit tends to raise an action’s probability and negative credit tends to lower it” only holds locally and is not generally correct, as parameters are shared, so this point is unclear.
-
The contribution appears primarily empirical. Eq 2 measures the change in an action’s log-prob before and after an update. Previous work, like [1], already studies how training on one example affects the probabilities of other ones. Also, [2] uses the change in log-pob assigned to the correct target across interaction steps as an intermediate reward. Here, Eq. 2 measures the probability change after a parameter update. While i understabd these methods differ in what changes between measurements and how the resulting signal is used, but this connection need to discussed. The paper should explain what is new about their analysis and which findings add to existing work on probability changes, gradient interactions, and optimizer history.
[1] Learning Dynamics of LLM Finetuning. Yi Ren and Danica J. Sutherland.
[2] Intrinsic Credit Assignment for Long Horizon Interaction. Ilze Amanda Auzina, Joschka Strüber, Sergio Hernández-Gutiérrez, Shashwat Goel, Ameya Prabhu, and Matthias Bethge.
-
The main experiments contains only limited setting, i.e. Qwen3.5 0.8B trained with LoRA, and five updates per complete run, but it is not clear whether the findings hold for other models or later stages of training. This matters because some of the reported behavior depends on initialization and the increasing contribution of past gradients during the first few updates. The ALFWorld and WebShop experiments test momentum attenuation, and they do not repeat the detailed attribution analysis in those environments. Also, the experiments are not done in a carefully controlled setting. For example, the momentum experiment changes several things at once such as attenuating history reduces supporting and opposing contributions, and it also changes the update magnitude. Hence, it is difficult to attribute the results to the effect of optimizer history.
-
It is unclear when the observed probability reversals are harmful. Optimizing a model with shared parameters involves tradeoffs between examples, so an action moving against its assigned credit does not necessarily imply a problem with learning. Indeed, the controlled runs improve substantially despite frequent reversals. Although the authors acknowledge this , the paper does not show when reducing reversals would improve task performance. The deletion experiments also keep the observed Adam scaling fixed. They explain contributions to a particular update, but do not establish what would happen if the corresponding experiences were removed from training.
In summary, since this paper is primarily an empirical paper, it requires comprehensive experiments and controlled one. However, in its current form, the paper lacks those and only covers a very limited setting.
Dear Reviewer PRpf,
We thank you for recognizing the motivation, presentation, and evidence of the paper, and for two precise questions. We respond to each below.
Q1: Why is k=2 not compared under the controls in Section 3.2?
The cumulative sweep covers k∈{1,2,4,8,16,32} (Table 16), and k=2 restores 63.4% of reversals, between 47.3% at k=1 and 80.6% at k=4. The matched controls are costly: for each target, k, and arm, we search up to 20,000 norm-matched subsets, which took 93,478 forward evaluations for three cutoffs. We therefore chose k=1 (a single source), k=4 (several sources), and k=8 (above 90% recovery). The ordering top opposing > other opposing > supporting holds at every update and every seed for all three cutoffs (Tables 21–22), so adding k=2 would not change the comparison. Appendix D.2 now states this reason.
Q2: Why does Section 3.3 use 19 seeds instead of 21?
The 19 seeds are the intersection of two audited sets (Appendix E.1). Seed 10 ran update 1 only, so 21 seeds have all five updates. We froze the update-1 audit protocol on seeds 1–20 before running seeds 21–22, so the prompt was not tuned on later data. Table 1 pools updates 1–5 and therefore uses the seeds audited at every update: 1–9 and 11–20. The update-1 rates over all 20 seeds (96.4% and 66.6%; Table 26) match the 19-seed entries in Table 1 (97.2% and 66.5%), so the restriction does not bias the result. The Table 1 caption and Appendix E.1 now say this.
Comments from the other reviewers led to a rewritten Abstract, a new Figure 1 built from the controlled results, added baselines (Table 13) and deletion controls (Table 19), and a new Section 4; the earlier capping tables are in Appendix F.
If time permits, we would be grateful if you could also look at the revised PDF and share any further comments.