acceptodds
Under review as a conference paper at ICLR 2027

Gradient-Momentum Coupling: A Parameter-Space Proxy for Learning Progress

Abstract

Measuring learning progress is at the core of curiosity-driven exploration, which rewards an agent for going where its model is still learning. However, the abstract notion of learning progress is not directly measurable, and existing methods often derive it from the prediction error in the output space. This paper proposes Gradient-Momentum Coupling (GMC), which measures how strongly a sample drives change in the parameter space, given by the normalized absolute product of its gradient with the momentum of previous gradients. Directions of change that persist across samples accumulate in momentum, while noise cancels out. In controlled experiments GMC allocates near uniform priority across tasks with varying levels of noise, where prediction error chases the noisiest, and orders learnable tasks by improvement speed rather than difficulty. On four MiniGrid MultiRoom tasks, substituting GMC for prediction error inside the Intrinsic Curiosity Module (ICM) recovers exploration ICM loses to unpredictable observations.

Primary area:
reinforcement learning
Keywords:
intrinsic motivation, curiosity-driven exploration, reinforcement learning, learning progress, aleatoric noise

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_qgZPOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes an alternative mechanism for learning progress that focuses on whether gradient updates are aligned in the same direction over time. This amounts to an attractive, yet simple, idea of using the dot product of the gradient with the momentum direction (although what is experimented on is different). While making learning progress one would expect the gradient to align with the momentum term, and while chasing noise as a signal, momentum and the gradient would be uncorrelated adressing the "noisy tv" problem in learning signals. Experimental results gives an illustration in a supervised learning setting, and then shows better return in a particular set of MiniGrid environments that grow in (exploration?) difficulty.

Critical strengths

Idea is simple and therefore attractive. Intuitively it addresses some of the failure cases of a learning progress metric (e.g., noisy TV, hard but learnable signals). The MiniGrid results are interesting.

Critical weaknesses

While it starts from a nice idea, that idea is not particularly well explored or developed.

The main idea is first enshrouded by a switcheroo from a principled dot-product of the gradient with the direction of the momentum, to a per-parameter product normalized by an EMA of the squared gradient. There is little justification for this in the main body of the paper, with deferrals of justification and explanation to the appendix (which still is not fully satisfying).

The experimental description and conclusions are drowned in dense prose that is common of LLM-generated text that invents jargon mid-paragraph, and seems to say a lot, while saying very little. the conclusions are also not readily available from the figures, oftne requiring the reader to try to decipher what they should be looking for. In the end, there's easily 4 pages saying what should take up 1 page.

And that suggests a lack of a really thorough exploration of the technique; and none of these experiments are particularly compute-intensive, suggesting a more thorough investigation is easily possible. While the experiments include baselines, they are all author-provided baselines with no connection to the cited work at all, which is disappointing and obscures the paper's contribution.

The two experiments don't seem to be the right ones. Where's the experiment with a controlled RL environment (some portion of the state space is aleatoric noise, other predictable noise) and show the agent's state visitation distribution over time? Where's a broad comparison of standard intrinsic reward methods (count-based, RND, etc.)?

Minor Comments / Suggestions:

L22 "... substituting GMC for prediction error ... recovers exploration ICM loses to unpredictable observations." This sentence barely parses. How do you recover exploration? Maybe "demonstrates the ability to explore even under unpredictable observations where ICM fails."

L28 What are "deceptive rewards"? I'm guessing you mean distractor rewards (which is the more typical name)?

L28 "address this"; what's this?

L34 "In the approaches that follow": that followed Schmidhuber's paper or that follow in this paragraph?

L54 I complete agree with this last sentence, but think the point could be made much more clearly.

L63 GMC is not defined in the main body yet.

L69 The italicized task performance appears nowhere else in the body of the paper (outside this paragraph), suggesting it's not really needed to be defined.

L91 "GMC scores learning on whichever representation these methods produce..."; malformed sentence.

L115 Why must \Delta_m be small? Because \beta is typically close to 1? Or (g-m) typically has a small magnitude? Neither seem always true.

L120 I think (1) would be a lot more clearly seen as the dot product between the direction of the momentum (m/||m||) and g.

L127 Isn't Equation (1) a single scalar (not a vector)?

L139 You keep deferring derivation and justification into the appendix, but it's starting to leave the motivation for you proposal extremely thin. I can understand (1) but by (3) I'm getting lost.

L148 Dropping back to measuring this on a separate model seems to be contradicting paragraph at L69. Or maybe I just didn't understand what that paragraph was trying to say.

L155 (1) What does "meaningfully different" mean? This criteria's not really objectively verifiable. (2) and (3) are more clear criteria. Is it enough for (3) to be substantiated with minigrid experiments?

L167 "Prioritized tasks": why are we prioritizing tasks? What does that mean? That seems non-standard for what is typical with intrinsic motivation signals.

L175 "This is analogous to an RL agent choosing which part of an environment to visit ..." This seems like a reach. There is much more that goes in to how agents take sequences of actions to reach parts of an environment! You realize this with the caveat at the end, "but without state or temporal credit assignment", but those two problems are the core of an "RL agent choosingwhich part of an environment to visit".

L194 "The four groups are the actions..." Why do you lead with this in Condition 2. Isn't also true in Condition 1. I'm left not really understanding these two conditions.

L200-215 Are these signals that are actually proposed in the literature? Citations? Do they correspond to signals covered by Linke et al.? Or are they just baselines for this set of experiments?

L228-239 Looks exactly like an LLM written paragraph. It aims to be precise but with no consideration for how human's read and comprehend. Hey, if I'm wrong, ignore this. But you'd be better off writing what you want to say rather than this not-wrong-but-still incomprehensible prose. Why is one y-axis label described in italics and the other in quotation marks?

Figure 1. Graphs are not in vector graphics format. There's no visible shaded region, which is probably because bounds are very tight, but that should be acknowledged.

L263-269 Some of the observations made in the text are not easy to identify what to look for in the graphs. The sentence starting "“Group size controls how many labels can be assigned, but it does not determine the empirical difficulty by itself..." is confusing.

L257 "Arrows mark each group’s visitation peak": The arrow at the blue line is clearly not a peak under Curiosity.

L281 "The MNIST (Appendix A) results show more pronounced effects due to larger difficulty differences between groups." Why wouldn't you show those results here then!?

L313-318 Again, I can't figure out how your prose is extracted from the plots. The ordering of the lines are the same for Curiosity and GMC. Is the difference I'm supposed to focus on that the red line grows more before flattening out?

L329 "This table reports failures in italics." What does this mean?!

L332 What is an "output-space error"?

L340 What does "from outside the model" mean?

L445-446 But these were experiments you just not to show. The only time you presented NormAll results in the main body of the paper was a table; so no over-time results can be concluded from the main body.

L456 "This supports the reading that GMC assumes no particular noise distribution": Does it? How?

Questions for authors

None.

3 comments
AuthorsOpenReview ↗

Addressing Weaknesses

Thank you for the detailed review and the many line-level comments. Most of them led to changes, listed in the next comment. We first address the main weaknesses.

  • Regarding "The two experiments don't seem to be the right ones...": The controlled setting, on which the main result is benchmarked, is comparable to a multi-armed bandit, a common abstraction of exploration in RL. Our setting takes it a few steps further by adding groups, observations, and noise. The setup closely follows Graves et al. 2017 (which works over tasks) and Matiisen et al. 2017 (both cited in the paper). It was chosen precisely because it isolates the signal itself, removing the potential confounds of credit assignment or policy optimization. Our controlled experiments are, in effect, the bandit form of the environment you describe. The allocation curves show the visitation distribution over time across groups that are either aleatoric noise or learnable. MultiRoom is the full-RL counterpart, and this is why it was chosen. The revision adds two manipulations closer to your request. In MultiRoom, giving ICM's forward model a history of observations makes part of the unpredictable residual predictable without changing the environment, which reduces ICM's rotation and raises its return from three rooms onward (Section 3.2). In the controlled setting, a Curriculum condition with label noise mixes learnable structure and aleatoric noise within the same tasks (Appendix A.3).

  • Regarding "Where's a broad comparison of ...": Our contribution is a learning-progress signal rather than a complete exploration algorithm, so we compare the raw signals such algorithms are built on. Section 3.1.2 enumerates these, each with a motivation sentence. Each serves to test a variation or a stepping stone along the way of constructing our method GMC. Section 4 discusses this choice, and Section 3.1.2 now points to it. The paper connected signals to related work in a scattered manner. We have now updated the paper to also contain citations for each signal to methods which build on them while each signal is defined. In the second part of the paper, the MultiRoom experiments show how our signal GMC can be applied in a full RL setup, and how it improves an existing algorithm, consistent with the controlled results. We show the fraction of rotations and forward moves, and the share of intrinsic reward given to rotations, and report in Section 3.2.2 how ICM without GMC "turns more and moves forward less, with the difference widening as the environments grow." Table 2 now gives these action shares for every method, putting concrete values on "less" and "widening", while the full curves remain in Appendix A.4 for space. Furthermore, MultiRoom now includes RND and ICM+LPM, and ICM+GMC stays ahead of both from three rooms onward (Section 3.2).

  • Regarding the motivation of Eq. 3: The motivation is outlined in the text. We first ask how to measure the contribution of one gradient vector to a momentum of gradient vectors (Eq. 1). We then apply this principle per parameter, which turns the vector product into a sum of scalar products. Normalizing by v gives each parameter equal weight, and the floor α^2 q on the denominator prevents a failure mode. That each parameter should have equal worth is indeed a design choice, and the experiments test it. The choice that the principle is applied per parameter / locally instead of on the full vectors is also a design choice, now compared against per-tensor variants in the main text (Section 2, Table 1). Both these choices are heuristic extensions of Eq. 1. Besides these choices, the Method section defers only the derivation of Eq. 1 and implementation details. Table 1 now contains the results of all methods, and shows GMC lower (better) loss AUC than or equal to the per-tensor Dot Product variant in all four columns. The differences are small, which suggests the per-parameter design choice suggests the per-parameter choice helps, although the margin is small.

AuthorsOpenReview ↗

Changes made for the minor comments

  • L22 Adopted your phrasing.
  • L28 "deceptive" Kept. "Sparse or deceptive rewards" seems to be the established phrase (e.g. Go-Explore, Ecoffet et al. 2019). Deceptive rewards lead toward local optima, which is a different problem from distractors.
  • L28 "address this" "This" refers to the challenge of efficient exploration in the preceding sentence.
  • L34 Changed to "that followed" to resolve ambiguity.
  • L54 Kept. We were unsure which part was unclear and would welcome a pointer.
  • L63 GMC is now defined at its first use in the main text.
  • L69 Kept, but rephrased to make explicit what we do not mean by "learning progress".
  • L91 "scores" -> "measures"
  • L115 We now write "generally small" since this usually holds in practice when the momentum approximates an average of recent gradients.
  • L120 We added the dot-product reading to the text and kept Eq. 1 unchanged to ease the transition to Eq. 3.
  • L127 Eq. 1 is a scalar. The sentence referred to the two vectors it compares, as opposed to the per-parameter comparison in Eq. 3, and is rephrased accordingly.
  • L139 The per-parameter choice is no longer deferred. The Method section now defines the per-tensor Dot Product and Cosine variants, and their results are in the main results table. Section 4 now uses the GMC-NoMom ablation to show which part of the construction matters (see also our main response).
  • L148 No contradiction is intended. L69 distinguishes learning progress from task performance. L148 states which network GMC is computed on, the dynamics model, which is also the model whose prediction error curiosity uses.
  • L155 (1) "Meaningfully" removed. (3) Yes, MultiRoom substitutes GMC into ICM within a full RL pipeline, now alongside RND, LPM and further controls.
  • L167 Section 1 now explains that automated curricula (Graves et al. 2017, Matiisen et al. 2017) prioritize tasks by learning progress, and how GMC differs by scoring individual samples without a predefined task partition.
  • L175 Kept. We agree these are the core problems of RL, which is why the analogy is limited to the choice between learning opportunities, which is what the intrinsic signal influences. Removing state and credit assignment is what isolates the signal, as the sentence states.
  • L194 Condition 1 states its actions a few sentences earlier ("The 10 reassigned classes are the actions"). Condition 2 now opens with "Here the actions are the four groups rather than the 10 classes" for clarity.
  • L200-215 Each signal now cites the work that uses it. Linke et al. cover three of them: squared error (Curiosity), Error Derivative (DeltaLoss) and Weight Change (related to NormAll). NormAll is also the gradient prediction gain of Graves et al. (2017), NormOut corresponds to the importance-sampling bound of Katharopoulos & Fleuret (2018), and DeltaLoss follows the same principle as TSCL and LPM.
  • L228-239 Both defined quantities are now in italics.
  • Fig. 1 We now note that some CIs are too narrow to be visible.
  • L263-269 Rewrote the group-size sentence, and replaced "peaks" with "prioritization order" throughout.
  • L257 Agreed. The caption and text now refer to when each group is prioritized rather than to a peak.
  • L281 Sentence removed to prevent confusion. We still prefer CIFAR-10 for the main text as they are more challenging and representative, and the MNIST results remain in Appendix A.8.
  • L313-318 Yes. Curiosity over-allocates to that group, which also raises its overall loss. The text now states this more directly.
  • L329 The table is replaced by the full results table, which has no italics.
  • L332 Rephrased to "the error in the model output space".
  • L340 Changed to "from the model outputs".
  • L445-446 The paragraph is rewritten around the new GMC-NoMom ablation, which isolates momentum better than NormAll. Its conclusions are now stated with numbers in the text, backed by Table 1 in the main body and by the per-condition table and curves in Appendix A.5.
  • L456 Agreed. Sentence removed.
Reviewer_qgZPOpenReview ↗

Response

While I appreciate the argument for the bandit experiments, I still don't think it does a very good job of qualitatively demonstrating how the approach works. To be clear, I'm not saying the bandit experiment holds no value, it just isn't sufficient given the motivations in the introduction and the fundamental nature of RL being about sequences of actions (i.e., being drawn to "noisy transitions" sort of requires transitions).

I also don't fully understand the argument "Our contribution is a learning-progress signal rather than a complete exploration algorithm". Pseudo-counts and RND are no more than a learning-progress signal. They don't then do anything particularly fancy to incorporate the learning progress signal than provide an additive intrinsic reward. And even if the signals do need special treatment that would make for a useful comparison: replace the RND signal under its separate head and discount factor with GMC or use RND in place of GMC in a more vanilla intrinsic reward usage. In any case, the addition of at least the RND baseline is helpful.

I'm still not convinced "Is it enough for (3) to be substantiated with minigrid experiments?", particularly when the return results don't show a particularly clear (statistically or otherwise) advantage over ICM alone (or RND in most cases).

This ties in to the switcheroo making all of the claims feel very muddled. What exactly is GMC? That GMC-NoMom (which now seems to full remove the idea of a dot-product of the gradient with momentum) also is highly performative. In fact, I'm most interested in Table 1's result where DotProduct alone (the very motivation in the introduction) is not that far behind and so would love to see what GMC adds to it in the minigrid experiments. In the end, exactly what's working here seems more muddled now than the original submission (admittedly, though, I've given considerably less time to it).

A skim also suggests the dense prose with low content to word ratio remain.

All-in-all, I think the fundamental weaknesses that landed me at weak reject remain.

Reviewer_cTdHOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Gradient-Momentum Coupling (GMC) to estimate learning progress. GMC uses sample gradients and historical gradient momentum to compute an intrinsic reward. Controlled experiments show less oversampling of noisy tasks. On larger MiniGrid MultiRoom tasks, GMC improves performance over the tested ICM baseline.

Critical strengths

-Rewards based on prediction error can attract agents to random noise. The paper addresses this problem using gradients and momentum. This is a meaningful question for exploration.

-GMC replaces the intrinsic reward within ICM while retaining its predictive training objectives. Scoring uses statistics from previous batches. A sample therefore does not directly contribute to the historical statistics used to score itself.

-The controlled tasks separate fixed-label memorization from resampled-label noise. The main controlled experiments use 20 seeds, and the RL experiments use 50. GMC shows a clear improvement on the six-room task. Action-frequency diagnostics provide additional evidence about the behavioral difference.

Critical weaknesses

W1: Equation (3) can be written as r(x)=d−1/2∑iwi∣gi(x)∣r(x)=d^{-1/2}\sum_i w_i|g_i(x)|, where wi=∣m^i∣/siw_i=|\hat m_i|/s_i. These weights are shared across the batch. The score therefore uses weighted gradient magnitudes. The current gradient’s sign relative to momentum does not affect the score. However, NormAll differs from GMC in both the norm and normalization. This comparison does not isolate the benefit of momentum weighting.

W2: Table 5 uses the same intrinsic reward coefficient, 5×10−45\times10^{-4}, for ICM and GMC. This does not ensure comparable reward strength. Equivalent reward-scale calibration in RL is not clearly documented. GMC may also reduce the relative influence of large prediction errors. A square-root transformation of ICM’s reward could have a similar effect. This alternative is not tested. The results therefore do not isolate momentum-based noise filtering from these simpler explanations.

W3: Limited RL Baselines and Mechanism Validation. The RL experiments cover four variants of the same MultiRoom family. Baselines are limited to PPO and ICM. The paper discusses LPM as a noise-robust alternative, but does not compare against it. The rotation explanation is supported by action frequencies and reward allocation. However, the experiments do not directly manipulate observation noise or the forward model’s access to history. The causal role of unpredictable observations therefore remains untested.

Questions for authors

Q1.Can the authors isolate the contribution of momentum weighting with a matched ablation? Does GMC favor noise when noisy and learnable gradients affect the same parameters? Please vary their overlap and the noise magnitude. Report both reward ratios and sampling allocations.

Q2. Do the RL gains persist after matched reward normalization and equal-budget reward-coefficient tuning? Please add an ICM baseline using the square root of prediction error. Does GMC still improve returns and reduce excessive rotation?

Q3. Can the authors compare GMC with LPM on a variant with controlled observation noise? Can they also give the forward model observation history while keeping policy inputs unchanged? Does this reduce ICM’s rotation bias and narrow the performance gap to GMC?

2 comments
AuthorsOpenReview ↗

Addressing Questions

Thank you for the detailed questions, which directly shaped most of the new experiments. The paper evaluates a learning-progress signal rather than proposing an exploration algorithm to be benchmarked, which is why the submission focused on ablations of the signal instead of a broad set of exploration baselines. Your controls test exactly the alternative explanations for that signal, so we added them.

  • W1, Q1: Isolating momentum. Agreed. We added a new ablation, GMC-NoMom, which removes the momentum factor keeping the normalization (Table 1, Appendix A.5, and MultiRoom in Section 3.2). This revealed that most of GMC's behavior comes from the normalization. However, momentum makes the allocation closer to uniform once learning saturates, so the allocation becomes less dependent on the noise level (Section 4, Appendix A.5). Noisy and learnable gradients already share parameters in the Noise condition, since all groups are learned by one network. We also added a Curriculum condition with label noise (Appendix A.3) which places noise inside every learnable class at two noise rates. At 20% noise, GMC maintains equal prioritization order without noise. At 50% it concentrates on the same two first groups but never reaches the last two, because the second group is not yet saturated by the end of the training.

  • W2, Q2: Reward scale and shape. All MultiRoom intrinsic rewards are divided by their running RMS and clipped before the shared coefficient is applied, so reward strength is matched across methods. This was used in the submitted experiments but not documented, so it is now listed in Appendix A.7. We added the ICM (sqrt reward) baseline you suggested. It performs below ICM from three rooms onward and turns even more in three and four rooms (Section 3.2, Table 2), so a flatter reward shape neither reproduces GMC's returns nor reduces rotation.

  • W3, Q3: Baselines and the rotation mechanism. We added RND and ICM+LPM. RND stays below ICM+GMC from three rooms onward, and ICM+LPM does not reach the goal in four or six rooms (Section 3.2). As you suggested, ICM (history) gives the forward model the preceding observations and actions while the encoder, inverse model and policy stay unchanged. This reduces ICM's rotation, most clearly in six rooms (92% to 48%), and brings its return close to ICM+GMC in three and four rooms but not in six. This supports the causal role of unpredictable observations. ICM+GMC (history) remains at or above ICM (history) in every environment (Section 3.2, Table 2, Appendix A.4). Controlled noise is covered by the Noise condition, where DeltaLoss represents the loss-change principle that LPM builds on.

Thank you for taking the time to address my comments. Your clarifications resolved some of my earlier concerns. I do not have any follow-up questions and will raise my score.

Reviewer_cyweOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Gradient-Momentum Coupling (GMC), a proxy for learning progress based on the normalized, elementwise coupling between a sample's model gradient and historical gradient momentum. The underlying idea is that momentum accumulates persistent directions of change and reduces the influence of inconsistent gradients on intrinsic rewards, thereby mitigating the attraction of prediction-error curiosity to noise. The authors first evaluate the signal using controlled task-difficulty and label-noise experiments on MNIST and CIFAR-10, and then integrate GMC into ICM on four MiniGrid MultiRoom environments. The results show reduced oversampling of high-noise groups and improved exploration over standard ICM in the more difficult MultiRoom settings. The controlled experiments primarily support the observed prioritization behavior and noise robustness, rather than a general improvement in convergence speed.

Critical strengths

The method is simple and integrates naturally with an existing curiosity framework. GMC constructs its signal from model gradients and momentum without training an additional learning-progress prediction network. In MultiRoom, the authors retain the ICM architecture and training objective and replace only the intrinsic reward signal. This makes the practical value of the parameter-space signal within an established exploration framework easy to understand and evaluate. The experiments examine prioritization behavior and exploration outcomes in useful detail. The paper combines controlled experiments with end-to-end reinforcement learning, using 20 and 50 random seeds respectively for the main experiments and reporting confidence intervals. The appendix additionally studies alternative class partitions, rotation and forward-action behavior, GMC variants, and momentum-decay sensitivity. These analyses increase confidence in the reported phenomena, particularly the observation that standard ICM persistently favors turning in MultiRoom.

Critical weaknesses

The current experiments do not directly validate the central motivation that loss can miss ongoing learning progress. The introduction argues that useful representations may continue developing when loss changes little. However, the controlled experiments mainly test noise robustness and prioritization order: GMC's ordering largely agrees with DeltaLoss, and the results do not show a general convergence advantage. The authors explicitly acknowledge that the experiments do not exercise a decoupling between loss and learning. I suggest adding one targeted controlled experiment, for example a phase in which predictive loss changes little while representation or transfer performance continues to improve, comparing GMC with DeltaLoss and Uniform and evaluating the usefulness of its prioritization through an independent downstream metric. This would directly support the introductory motivation; otherwise, the claims about going beyond loss-change signals should be narrowed accordingly. More direct end-to-end controls would help separate momentum coupling, normalization, and reward-scale effects. NormAll removes both momentum weighting and per-parameter normalization, so its difference from GMC cannot be attributed to momentum alone. The Dot Product and Cosine variants retain both momentum and normalization and therefore do not resolve this question. Furthermore, the controlled experiments use early signal-scale calibration, whereas the MultiRoom setup reports a shared intrinsic reward coefficient for ICM and GMC; an identical coefficient need not imply comparable effective reward magnitudes. In one representative MultiRoom setting, I suggest adding a gradient-magnitude control that retains the same normalization but removes historical momentum coupling, together with either early scale matching or a small intrinsic-reward coefficient comparison to test whether the return advantage persists.

Questions for authors

Are there experimental results for a setting in which loss and learning progress decouple? If not, could the scope of the present conclusions be stated more explicitly? How do prioritization behavior and MultiRoom returns change when GMC's per-parameter normalization is retained but historical momentum coupling is removed? Was signal-scale calibration also applied in MultiRoom, or were the raw signals multiplied by the same coefficient? Does GMC retain its advantage over ICM under scale matching or nearby intrinsic-reward coefficients?

2 comments
AuthorsOpenReview ↗

Addressing questions

We have now added the momentum control you proposed, and the general overview lists all other changes.

  • Decoupling of loss and learning progress (W1, Q1). We have no experiment in which loss and learning progress decouple, so we narrowed the scope as you suggested. The motivation in Section 1 now rests on prior work showing that internal structure can keep developing while the loss is flat or after it has vanished. The Limitations (Section 4) state that our controlled experiments do not exercise this decoupling, since GMC's ordering there coincides with that of DeltaLoss. Our experimental claims are therefore limited to noise robustness, prioritization behavior, and exploration in MultiRoom.

  • Separating momentum from normalization (W2, Q2). We added the control you proposed. GMC-NoMom keeps GMC's per-parameter normalization and removes only the momentum, and it is evaluated in the controlled experiments (Table 1, Appendix A.5) and in MultiRoom (Section 3.2, Table 2, Appendix A.4). It shows that most of GMC's behavior comes from the normalization. GMC-NoMom reproduces GMC's curriculum, its loss AUC, and its advantage over ICM in MultiRoom. Momentum's role is narrower, it keeps the allocation closer to uniform once learning saturates, which is most visible in the Noise condition. Section 4 now presents GMC in these terms.

  • Reward scale in MultiRoom (W2, Q3, Q4). In MultiRoom the intrinsic reward of every method is divided by its running RMS and clipped before the shared coefficient is applied, so the effective reward magnitudes are matched. This was used in the submitted experiments but not documented, and it is now listed in Appendix A.7. As an additional control we added ICM (sqrt reward), which, like GMC, reduces the weight of large prediction errors. It performs below ICM from three rooms onward and rotates even more often in three and four rooms (Section 3.2, Table 2), so GMC's advantage is not explained by a flatter reward shape.

Thank you for adding the GMC-NoMom ablation, clarifying reward normalization, and narrowing the claims. Thus, I will keep my rating of 3.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of changes in revised paper

We thank all reviewers for their feedback. Below is a short overview of the main changes in the revised paper.

New experiments

  • GMC-NoMom ablation, which keeps GMC's per-parameter normalization and removes only the momentum, in both the controlled experiments (Section 3.1, Appendix A.5) and MultiRoom (Section 3.2, Appendix A.4).
  • New MultiRoom baselines and controls (Section 3.2, Appendix A.4): RND, ICM+LPM, ICM (sqrt reward) as a reward-shape control, and ICM and ICM+GMC with an observation history in the forward model. These use 10 seeds for now. Runs with 50 seeds are in progress and will be included in the camera-ready version. At 10 seeds the confidence intervals are already narrow enough to support the reported conclusions.
  • Curriculum condition with label noise, combining task difficulty and noise in a single setting (Appendix A.3).

Results, interpretation and related work

  • The main text now reports the loss AUC of all methods and GMC variants on both datasets (Table 1), and the end-of-training action shares of all MultiRoom methods (Table 2), which were previously reported only in the appendix.
  • Following the GMC-NoMom results, the discussion now attributes most of GMC's behavior to its per-parameter second-moment normalization, with momentum mainly reducing the dependence of the allocation on the noise level once learning saturates (Section 4, Appendix A.5).
  • Narrowed claims: we state explicitly that our experiments do not test a decoupling between loss and learning, and motivate it with prior work instead. Added references showing that a model's internal structure can keep developing while its loss is flat or after it has vanished (Section 1).
  • Each baseline signal of the controlled experiments is now linked to its counterpart in the literature (Section 3.1), and the relation to learning-progress curricula is clarified (Section 1, Relation to Prior Work).

Method and reproducibility

  • Equation 1 is now presented as a dot product with the momentum direction, the role of the second moment is now introduced in the motivation (Section 1), and the Dot Product and Cosine variants are defined in the main text (Section 2).
  • The MultiRoom setup now documents the running intrinsic-reward normalization shared by all methods, and the settings of the new baselines (Appendix A.7).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.