- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes to split the step size T of the policy update during the policy improvement step in offline RL, and take K repeated gradient steps instead, of size T/K each. The motivation is that repeated gradient steps can reach different points in function space during optimization, compared to a single step with a large step size.
Critical strengths
-
The idea is rather simple and straightforward.
-
The presentation is, for the most part, clear.
-
Results could be promising.
Critical weaknesses
Motivation, novelty and significance:
The motivation for the approach, as I understand it, is that multiple smaller updates can arrive at different points during optimization, compared to a single update. This is a well known property to my knowledge and goes beyond regularization (which is the main point of view taken by this work). The approach proposed in this work, as I understand it, is simply to do multiple smaller updates. This idea is, again, not novel (for example, see [1]), and seems to me to be effectively but another way to increase the replay-ratio (the number of updates per iteration, as opposed to the size of each update). This reduces the novelty of the approach singificantly. Additionally, increased replay ratio is not discussed or compared to I believe, which prevents me from being able to evaluate the significance of this contribution.
Evaluation, its fairness, and its presentation:
The method, unless I misunderstand it (in which case, please correct me), comprises keeping K policy heads / individual DNNs (line 160), and training them with respect to each other. This raises multiple questions which I think are major and are not addressed by the paper (unless I missed them - in which case, again please correct me):
-
Why keep K actors, and not just train the same actor regularized with respect to itself with stop-gradient?
-
What is the increased cost, and is it worse it?
- First, the cost in parameters from maintaining K - 1 additional actors, and isn't it better to spend the same parameters increasing the critic / standard actor's network sizes?.
- Second, the cost in number of updates (The "replay ratio"). K updates have approximate times-K the cost of a single update. Is this tradeoff every worthwhile? From the results, it seems to me the answer is strongly no, but an explicit experiment is critical here in my opinion, either way.
-
It's not clear to me what exactly is being evaluated in the experiment when T is increased. I see two possibilities, both are problematic:
- T is effectively the learning rate of the baseline, and the I-MPI-K does K updates of size T/K. If this is the case: Is it even meaningful to increase T? T is presented as a notion of horizon, but unless I misunderstand (in which case as usual please correct me) its much more prudent to think of it as a learning rate (Figure 1). Decades (more?) of experience in ML tell us that changing the learning rate can cause almost arbitrary decay in performance, and that tuning learning rate is super important. So this makes comparisons to the baseline close to meaningless, in my opinion.
- T decides the size of the steps for I-MPI-K. In which case, the comparison to the baseline is unfair, as its strictly getting more updates? And how does this correlate with tuning learning rate?
-
Tables are the least-preferable way to visualize data. I strongly recommend changing Table 1 to something more visually informative - perhaps a bar plot?
Statistical significance:
Missing from many results (Figures 2, 5, and 6, Table 1).
Inappropriate stat. signif. metric for the other results (Figures 3 and 4). Please replace the single standard-dev. over seeds with a proper confidence bound (for example, 95% Gaussian CI, e.g. 2xStandard error of the mean, when eveluating the significance of the mean performance).
[1] Oren et. al., Value Improved Actor Critic Algorithms. NIPS 2025.
Questions for authors
Please see the weaknesses section.
I thank the authors for their more detailed rebuttal. Unfortunately, it is still phrased in a way that is difficut for me to understand how, if, and to what extent it addresses the concerns I've raised.
I would like to explicitly restate the following:
Soundness:
It remains unclear to me what is the fundemental advantage of this approach, and if there's reason to expect it to be worth the compute cost. This is not to suggest that there isn't any - it is merely that it remains unclear to me from the current presentation of the paper. I encourage the authors to iterate on this in the next version of the document.
Statistical significance:
Every result presented in the paper needs to be paired with the appropriate metric for statistical significance. Table 1 is still not paired with statistical significance. Results are close to meaningless without statistical significance. The reader does not need to go to the appendix to see it.
Presentation and results:
In my understanding, the most important comparisons for a method that increases the number of updates is comparison to other increased-update methods (e.g. increased replay ratio), as well as wallclock runtime. I believe sufficient comparisons of this kind are still missing.
In addition, I find the presentation of the results a bit unclear. I think the main reason is that the author's approach is not entirely standard (at least, not that I've seen). That's not a bad thing - it suggests novelty. However, it makes the results harder to interpret, and thus appreciate the authors' contribution. I think the method and the experiment section will benefit from clearer presentation.
For these reasons, I am unable to change my score.