- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies whether reinforcement learning (RL) can elicit solutions to tasks not explicitly targeted during pre-training. Specifically, the authors study a controlled setting on dense parity functions where pre-training uses a mixture of short-answer demonstrations and rare CoT-like sequences, while post-training targets a different parity function. Empirically, they show that GRPO can achieve nontrivial accuracy on the new task, whereas supervised fine-tuning (SFT) fails. They further observe that RL training often exhibits phase transitions and suboptimal accuracy plateaus. To explain these phenomena, the authors develop a theoretical framework suggesting that RL can learn low-dimensional features that identify a subset of input space where pretrained computations can be reused.
Critical strengths
-
This paper discusses an important question of whether RL post-training can enable language models to learn new tasks beyond those in pre-training.
-
The logic chain of the empirical analysis is clear and easy to follow. The authors show step by step that (a) RL effectively adapts to unseen parities with accuracy above 50% (usually 75% or 100%). (b) This adaptivity arises from learning low-dimensional features: a gate characterizing the difference between the pretraining target and posttraining target. The explanation makes sense to me.
-
The author provides a theoretical account of how RL post-training enables transfer between dense parity targets.
Critical weaknesses
Overall, I think this paper is interesting, and I give it a positive rating. The main reasons for not giving a higher score are as follows:
-
As the authors admitted in the conclusion section, the theoretical setting is too simplified. The theoretical analysis considers an idealized STaR objective with linear autoregressive models, while the experiments use GRPO with Transformers. Moreover, the theory assumes optimal policy updates without analyzing the actual optimization process.
-
The interpretation of acquiring new capabilities may be overstated. In my view, learning a low-dimensional feature to identify input regions where the pre-training and post-training targets agree does not necessarily constitute a meaningful new reasoning capability. Instead, it can be viewed as an undesirable shortcut that achieves partial accuracy without genuinely learning the new parity function. While understanding this shortcut is interesting, I believe it would be more important to investigate why RL converges to such suboptimal solutions and how it can be encouraged to learn the correct parity function instead.
Questions for authors
Please see above.
3 comments
I appreciate the authors' efforts to address my concerns.
The new GRPO analysis in Appendix I and the additional experiments on increased sampling temperature provide useful insights and partially address my concerns. While the theoretical analysis still relies on a simplified setting and does not fully capture the optimization dynamics of Transformers, I understand that developing a more general theoretical characterization is challenging and may be beyond the scope of the rebuttal period.
The observation that increased exploration can help avoid suboptimal shortcut strategies makes sense to me. Intuitively, exploration is a useful method to escape local minima. Also,maybe modifying the post-training data, for example by introducing greater diversity in CoT formats or reasoning traces, could reduce the tendency to learn such shortcuts in the first place. There has been related work exploring the role of data diversity in other tasks and settings. This might offer a more data-centric perspective on the problem. I understand that this would likely constitute a separate research question, and I do not ask the authors to investigate it as part of the current work. I simply share this thought here.
Overall, I appreciate the authors' thoughtful response and the additional efforts made in the revision. I remain supportive of accepting this paper.
Thank you for your reply and for sharing your thoughts on exploration.
Thank you for your review. We are pleased you found our paper interesting and the analysis clear and easy to follow. We respond to your points:
We believe that our analysis takes a necessary first step. As prior results on reinforcement learning of autoregressive networks are scarce, we believe it is natural to first understand how RL changes the dynamics in an architecture-agnostic way and then attempt to obtain a finer-grained understanding. As we showed in our analysis, this already offers a mathematically rich landscape with unique phenomena worth understanding (e.g., the form of the shifted distributions over the RL rounds).
Furthermore, in response to your question and that of reviewer 34vr, we added a section to the appendix (Appendix I) and a reference to it in the main text, where we demonstrate that it is possible to extend the analysis to cover the GRPO objective. Assuming the same linear autoregressive policy, we show that one population GRPO update introduces agreement-dependent behavior at the first output position. This implies that the GRPO objective similarly encourages the formation of the low-dimensional feature gk and could similarly lead to shortcut solutions. While we did not want to rush to include a full end-to-end result for GRPO, we believe that Appendix I, and Proposition 7 in particular, show that this might in fact be possible. We also explain there a promising way to achieve this (i.e., by considering the limit points of gradient flow, which is quite different from the STaR analysis). It would indeed be interesting for future work to study this and perhaps obtain a fine-grained understanding of the duration of the suboptimal plateaus observed in the experiments, and whether these depend more strongly on the loss. Understanding optimization dynamics of RL with transformers remains an important outstanding problem.
Thank you for the suggestion. Following your suggestion, we experimented with a few different strategies that could prevent the model from converging to a suboptimal strategy, and we found that exploration, in the form of increased temperature, can help the model reach strategies that achieve perfect generalization and consistently output "faithful" reasoning. In particular, in Figure 11 (Appendix B), we show that increasing the RL sampling temperature leads to many more random seeds converging to perfect accuracy, often doubling the chances of this happening compared to the temperature 1.0 baseline. We believe this provides additional evidence supporting similar remedies that have been proposed in the literature on frontier LLMs. Please consider taking a look at the paragraph titled "Exploration helps prevent shortcuts" in Appendix B, and let us know if you have any questions.