- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes an alternative mechanism for learning progress that focuses on whether gradient updates are aligned in the same direction over time. This amounts to an attractive, yet simple, idea of using the dot product of the gradient with the momentum direction (although what is experimented on is different). While making learning progress one would expect the gradient to align with the momentum term, and while chasing noise as a signal, momentum and the gradient would be uncorrelated adressing the "noisy tv" problem in learning signals. Experimental results gives an illustration in a supervised learning setting, and then shows better return in a particular set of MiniGrid environments that grow in (exploration?) difficulty.
Critical strengths
Idea is simple and therefore attractive. Intuitively it addresses some of the failure cases of a learning progress metric (e.g., noisy TV, hard but learnable signals). The MiniGrid results are interesting.
Critical weaknesses
While it starts from a nice idea, that idea is not particularly well explored or developed.
The main idea is first enshrouded by a switcheroo from a principled dot-product of the gradient with the direction of the momentum, to a per-parameter product normalized by an EMA of the squared gradient. There is little justification for this in the main body of the paper, with deferrals of justification and explanation to the appendix (which still is not fully satisfying).
The experimental description and conclusions are drowned in dense prose that is common of LLM-generated text that invents jargon mid-paragraph, and seems to say a lot, while saying very little. the conclusions are also not readily available from the figures, oftne requiring the reader to try to decipher what they should be looking for. In the end, there's easily 4 pages saying what should take up 1 page.
And that suggests a lack of a really thorough exploration of the technique; and none of these experiments are particularly compute-intensive, suggesting a more thorough investigation is easily possible. While the experiments include baselines, they are all author-provided baselines with no connection to the cited work at all, which is disappointing and obscures the paper's contribution.
The two experiments don't seem to be the right ones. Where's the experiment with a controlled RL environment (some portion of the state space is aleatoric noise, other predictable noise) and show the agent's state visitation distribution over time? Where's a broad comparison of standard intrinsic reward methods (count-based, RND, etc.)?
Minor Comments / Suggestions:
L22 "... substituting GMC for prediction error ... recovers exploration ICM loses to unpredictable observations." This sentence barely parses. How do you recover exploration? Maybe "demonstrates the ability to explore even under unpredictable observations where ICM fails."
L28 What are "deceptive rewards"? I'm guessing you mean distractor rewards (which is the more typical name)?
L28 "address this"; what's this?
L34 "In the approaches that follow": that followed Schmidhuber's paper or that follow in this paragraph?
L54 I complete agree with this last sentence, but think the point could be made much more clearly.
L63 GMC is not defined in the main body yet.
L69 The italicized task performance appears nowhere else in the body of the paper (outside this paragraph), suggesting it's not really needed to be defined.
L91 "GMC scores learning on whichever representation these methods produce..."; malformed sentence.
L115 Why must \Delta_m be small? Because \beta is typically close to 1? Or (g-m) typically has a small magnitude? Neither seem always true.
L120 I think (1) would be a lot more clearly seen as the dot product between the direction of the momentum (m/||m||) and g.
L127 Isn't Equation (1) a single scalar (not a vector)?
L139 You keep deferring derivation and justification into the appendix, but it's starting to leave the motivation for you proposal extremely thin. I can understand (1) but by (3) I'm getting lost.
L148 Dropping back to measuring this on a separate model seems to be contradicting paragraph at L69. Or maybe I just didn't understand what that paragraph was trying to say.
L155 (1) What does "meaningfully different" mean? This criteria's not really objectively verifiable. (2) and (3) are more clear criteria. Is it enough for (3) to be substantiated with minigrid experiments?
L167 "Prioritized tasks": why are we prioritizing tasks? What does that mean? That seems non-standard for what is typical with intrinsic motivation signals.
L175 "This is analogous to an RL agent choosing which part of an environment to visit ..." This seems like a reach. There is much more that goes in to how agents take sequences of actions to reach parts of an environment! You realize this with the caveat at the end, "but without state or temporal credit assignment", but those two problems are the core of an "RL agent choosingwhich part of an environment to visit".
L194 "The four groups are the actions..." Why do you lead with this in Condition 2. Isn't also true in Condition 1. I'm left not really understanding these two conditions.
L200-215 Are these signals that are actually proposed in the literature? Citations? Do they correspond to signals covered by Linke et al.? Or are they just baselines for this set of experiments?
L228-239 Looks exactly like an LLM written paragraph. It aims to be precise but with no consideration for how human's read and comprehend. Hey, if I'm wrong, ignore this. But you'd be better off writing what you want to say rather than this not-wrong-but-still incomprehensible prose. Why is one y-axis label described in italics and the other in quotation marks?
Figure 1. Graphs are not in vector graphics format. There's no visible shaded region, which is probably because bounds are very tight, but that should be acknowledged.
L263-269 Some of the observations made in the text are not easy to identify what to look for in the graphs. The sentence starting "“Group size controls how many labels can be assigned, but it does not determine the empirical difficulty by itself..." is confusing.
L257 "Arrows mark each group’s visitation peak": The arrow at the blue line is clearly not a peak under Curiosity.
L281 "The MNIST (Appendix A) results show more pronounced effects due to larger difficulty differences between groups." Why wouldn't you show those results here then!?
L313-318 Again, I can't figure out how your prose is extracted from the plots. The ordering of the lines are the same for Curiosity and GMC. Is the difference I'm supposed to focus on that the red line grows more before flattening out?
L329 "This table reports failures in italics." What does this mean?!
L332 What is an "output-space error"?
L340 What does "from outside the model" mean?
L445-446 But these were experiments you just not to show. The only time you presented NormAll results in the main body of the paper was a table; so no over-time results can be concluded from the main body.
L456 "This supports the reading that GMC assumes no particular noise distribution": Does it? How?
Questions for authors
None.
3 comments
Changes made for the minor comments
- L22 Adopted your phrasing.
- L28 "deceptive" Kept. "Sparse or deceptive rewards" seems to be the established phrase (e.g. Go-Explore, Ecoffet et al. 2019). Deceptive rewards lead toward local optima, which is a different problem from distractors.
- L28 "address this" "This" refers to the challenge of efficient exploration in the preceding sentence.
- L34 Changed to "that followed" to resolve ambiguity.
- L54 Kept. We were unsure which part was unclear and would welcome a pointer.
- L63 GMC is now defined at its first use in the main text.
- L69 Kept, but rephrased to make explicit what we do not mean by "learning progress".
- L91 "scores" -> "measures"
- L115 We now write "generally small" since this usually holds in practice when the momentum approximates an average of recent gradients.
- L120 We added the dot-product reading to the text and kept Eq. 1 unchanged to ease the transition to Eq. 3.
- L127 Eq. 1 is a scalar. The sentence referred to the two vectors it compares, as opposed to the per-parameter comparison in Eq. 3, and is rephrased accordingly.
- L139 The per-parameter choice is no longer deferred. The Method section now defines the per-tensor Dot Product and Cosine variants, and their results are in the main results table. Section 4 now uses the GMC-NoMom ablation to show which part of the construction matters (see also our main response).
- L148 No contradiction is intended. L69 distinguishes learning progress from task performance. L148 states which network GMC is computed on, the dynamics model, which is also the model whose prediction error curiosity uses.
- L155 (1) "Meaningfully" removed. (3) Yes, MultiRoom substitutes GMC into ICM within a full RL pipeline, now alongside RND, LPM and further controls.
- L167 Section 1 now explains that automated curricula (Graves et al. 2017, Matiisen et al. 2017) prioritize tasks by learning progress, and how GMC differs by scoring individual samples without a predefined task partition.
- L175 Kept. We agree these are the core problems of RL, which is why the analogy is limited to the choice between learning opportunities, which is what the intrinsic signal influences. Removing state and credit assignment is what isolates the signal, as the sentence states.
- L194 Condition 1 states its actions a few sentences earlier ("The 10 reassigned classes are the actions"). Condition 2 now opens with "Here the actions are the four groups rather than the 10 classes" for clarity.
- L200-215 Each signal now cites the work that uses it. Linke et al. cover three of them: squared error (Curiosity), Error Derivative (DeltaLoss) and Weight Change (related to NormAll). NormAll is also the gradient prediction gain of Graves et al. (2017), NormOut corresponds to the importance-sampling bound of Katharopoulos & Fleuret (2018), and DeltaLoss follows the same principle as TSCL and LPM.
- L228-239 Both defined quantities are now in italics.
- Fig. 1 We now note that some CIs are too narrow to be visible.
- L263-269 Rewrote the group-size sentence, and replaced "peaks" with "prioritization order" throughout.
- L257 Agreed. The caption and text now refer to when each group is prioritized rather than to a peak.
- L281 Sentence removed to prevent confusion. We still prefer CIFAR-10 for the main text as they are more challenging and representative, and the MNIST results remain in Appendix A.8.
- L313-318 Yes. Curiosity over-allocates to that group, which also raises its overall loss. The text now states this more directly.
- L329 The table is replaced by the full results table, which has no italics.
- L332 Rephrased to "the error in the model output space".
- L340 Changed to "from the model outputs".
- L445-446 The paragraph is rewritten around the new GMC-NoMom ablation, which isolates momentum better than NormAll. Its conclusions are now stated with numbers in the text, backed by Table 1 in the main body and by the per-condition table and curves in Appendix A.5.
- L456 Agreed. Sentence removed.
Response
While I appreciate the argument for the bandit experiments, I still don't think it does a very good job of qualitatively demonstrating how the approach works. To be clear, I'm not saying the bandit experiment holds no value, it just isn't sufficient given the motivations in the introduction and the fundamental nature of RL being about sequences of actions (i.e., being drawn to "noisy transitions" sort of requires transitions).
I also don't fully understand the argument "Our contribution is a learning-progress signal rather than a complete exploration algorithm". Pseudo-counts and RND are no more than a learning-progress signal. They don't then do anything particularly fancy to incorporate the learning progress signal than provide an additive intrinsic reward. And even if the signals do need special treatment that would make for a useful comparison: replace the RND signal under its separate head and discount factor with GMC or use RND in place of GMC in a more vanilla intrinsic reward usage. In any case, the addition of at least the RND baseline is helpful.
I'm still not convinced "Is it enough for (3) to be substantiated with minigrid experiments?", particularly when the return results don't show a particularly clear (statistically or otherwise) advantage over ICM alone (or RND in most cases).
This ties in to the switcheroo making all of the claims feel very muddled. What exactly is GMC? That GMC-NoMom (which now seems to full remove the idea of a dot-product of the gradient with momentum) also is highly performative. In fact, I'm most interested in Table 1's result where DotProduct alone (the very motivation in the introduction) is not that far behind and so would love to see what GMC adds to it in the minigrid experiments. In the end, exactly what's working here seems more muddled now than the original submission (admittedly, though, I've given considerably less time to it).
A skim also suggests the dense prose with low content to word ratio remain.
All-in-all, I think the fundamental weaknesses that landed me at weak reject remain.
Addressing Weaknesses
Thank you for the detailed review and the many line-level comments. Most of them led to changes, listed in the next comment. We first address the main weaknesses.
Regarding "The two experiments don't seem to be the right ones...": The controlled setting, on which the main result is benchmarked, is comparable to a multi-armed bandit, a common abstraction of exploration in RL. Our setting takes it a few steps further by adding groups, observations, and noise. The setup closely follows Graves et al. 2017 (which works over tasks) and Matiisen et al. 2017 (both cited in the paper). It was chosen precisely because it isolates the signal itself, removing the potential confounds of credit assignment or policy optimization. Our controlled experiments are, in effect, the bandit form of the environment you describe. The allocation curves show the visitation distribution over time across groups that are either aleatoric noise or learnable. MultiRoom is the full-RL counterpart, and this is why it was chosen. The revision adds two manipulations closer to your request. In MultiRoom, giving ICM's forward model a history of observations makes part of the unpredictable residual predictable without changing the environment, which reduces ICM's rotation and raises its return from three rooms onward (Section 3.2). In the controlled setting, a Curriculum condition with label noise mixes learnable structure and aleatoric noise within the same tasks (Appendix A.3).
Regarding "Where's a broad comparison of ...": Our contribution is a learning-progress signal rather than a complete exploration algorithm, so we compare the raw signals such algorithms are built on. Section 3.1.2 enumerates these, each with a motivation sentence. Each serves to test a variation or a stepping stone along the way of constructing our method GMC. Section 4 discusses this choice, and Section 3.1.2 now points to it. The paper connected signals to related work in a scattered manner. We have now updated the paper to also contain citations for each signal to methods which build on them while each signal is defined. In the second part of the paper, the MultiRoom experiments show how our signal GMC can be applied in a full RL setup, and how it improves an existing algorithm, consistent with the controlled results. We show the fraction of rotations and forward moves, and the share of intrinsic reward given to rotations, and report in Section 3.2.2 how ICM without GMC "turns more and moves forward less, with the difference widening as the environments grow." Table 2 now gives these action shares for every method, putting concrete values on "less" and "widening", while the full curves remain in Appendix A.4 for space. Furthermore, MultiRoom now includes RND and ICM+LPM, and ICM+GMC stays ahead of both from three rooms onward (Section 3.2).
Regarding the motivation of Eq. 3: The motivation is outlined in the text. We first ask how to measure the contribution of one gradient vector to a momentum of gradient vectors (Eq. 1). We then apply this principle per parameter, which turns the vector product into a sum of scalar products. Normalizing by v gives each parameter equal weight, and the floor α^2 q on the denominator prevents a failure mode. That each parameter should have equal worth is indeed a design choice, and the experiments test it. The choice that the principle is applied per parameter / locally instead of on the full vectors is also a design choice, now compared against per-tensor variants in the main text (Section 2, Table 1). Both these choices are heuristic extensions of Eq. 1. Besides these choices, the Method section defers only the derivation of Eq. 1 and implementation details. Table 1 now contains the results of all methods, and shows GMC lower (better) loss AUC than or equal to the per-tensor Dot Product variant in all four columns. The differences are small, which suggests the per-parameter design choice suggests the per-parameter choice helps, although the margin is small.