- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The gives theoretical results regarding an asymmetry in the convergence rates of actor-critic in RL: the critic converges exponentially while the actor converges linearly. They propose to leverage this by updating the critic less often and thus increasing training efficiency. They provide experiments.
Critical strengths
- The empirical observation that the critic converges faster than the actor, and that this can be leveraged to improve training efficiency, in the context of llm post training is, to the best of my knowledge, both novel and relevant for practitioners.
Critical weaknesses
-
The theoretical results do not justify the asymmetry but rather assume it. That is, the convergence rates are standard/general results that stem from the PL assumption in the case of the critic and from the smoothness in the case of the actor. In my understanding the same results apply to any training objective satisfying these assumptions and has nothing to do with actor-critic. Therefore, the paper fails to address where this assymetry in actor and critic convergence stems from, but rather presents assumptions that imply it, rendering most results tautological. If the theoretical analysis of assymetry in actor critic is a major contribution of the manuscript, and the theory plays such an important role in its presentation, then why the PL condition would hold for the critic (which is deferred to an appendix) but not for the actor (as far as I know, missing from the paper) should be justified and discussed in the main body.
-
Given their popularity, I would find the inclusion of at least one critic-free method in the experimental section pertinent
-
Regarding presentation, I think that the description of the proposed method intertwined with the theoretical results lacks clarity. Having a separate proposed method / algorithmic section in the body could help.
-
All of the experiment results in section 5 all focus on accuracy vs flops, whereas the main result pertains actor and critic convergence, so some additional convergence plots (apart from the one in the first page) and analysis could be helpful.
Questions for authors
How do you justify the assymetry in assumptions for the critic and actor and how these relate to their learning objectives (see weakness 1)?