acceptodds
Under review as a conference paper at ICLR 2027

From Target Shift to Covariate Shift: How RL Post-Training Enables Transfer to New Tasks

Abstract

Reinforcement learning (RL) post-training has substantially improved the reasoning capabilities of autoregressive language models, but whether it can elicit solutions to tasks not explicitly targeted during pre-training remains unclear. We study this question through the canonical problem of learning parities over uniform input bits. Pre-training data consist primarily of short answers to a parity task, together with rare chain-of-thought (CoT) demonstrations that include intermediate computations. During RL post-training, the reward instead targets a distinct parity function that is uncorrelated with the pre-training target. Experimentally, we observe that RL post-training can achieve nontrivial accuracy on the new task, escaping apparent computational barriers, while supervised fine-tuning under the same information is ineffective. During the course of RL training, networks undergo sudden phase changes in both accuracy and generation length, and can converge to suboptimal strategies. By developing an effective theory of the RL dynamics, we characterize some of these suboptimal strategies, and link them to the formation of low-dimensional features that can be learned efficiently, yet are predictive of the new target over only some parts of the input space. We explain how the RL loss encourages the discovery of these features by turning the target shift into a covariate shift, allowing the model to repurpose pretrained CoTs on inputs where the two targets coincide.

Primary area:
learning theory
Keywords:
rl post-training, new skills, transfer learning, parities

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zez1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies whether reinforcement learning (RL) can elicit solutions to tasks not explicitly targeted during pre-training. Specifically, the authors study a controlled setting on dense parity functions where pre-training uses a mixture of short-answer demonstrations and rare CoT-like sequences, while post-training targets a different parity function. Empirically, they show that GRPO can achieve nontrivial accuracy on the new task, whereas supervised fine-tuning (SFT) fails. They further observe that RL training often exhibits phase transitions and suboptimal accuracy plateaus. To explain these phenomena, the authors develop a theoretical framework suggesting that RL can learn low-dimensional features that identify a subset of input space where pretrained computations can be reused.

Critical strengths
  1. This paper discusses an important question of whether RL post-training can enable language models to learn new tasks beyond those in pre-training.

  2. The logic chain of the empirical analysis is clear and easy to follow. The authors show step by step that (a) RL effectively adapts to unseen parities with accuracy above 50% (usually 75% or 100%). (b) This adaptivity arises from learning low-dimensional features: a gate characterizing the difference between the pretraining target and posttraining target. The explanation makes sense to me.

  3. The author provides a theoretical account of how RL post-training enables transfer between dense parity targets.

Critical weaknesses

Overall, I think this paper is interesting, and I give it a positive rating. The main reasons for not giving a higher score are as follows:

  1. As the authors admitted in the conclusion section, the theoretical setting is too simplified. The theoretical analysis considers an idealized STaR objective with linear autoregressive models, while the experiments use GRPO with Transformers. Moreover, the theory assumes optimal policy updates without analyzing the actual optimization process.

  2. The interpretation of acquiring new capabilities may be overstated. In my view, learning a low-dimensional feature to identify input regions where the pre-training and post-training targets agree does not necessarily constitute a meaningful new reasoning capability. Instead, it can be viewed as an undesirable shortcut that achieves partial accuracy without genuinely learning the new parity function. While understanding this shortcut is interesting, I believe it would be more important to investigate why RL converges to such suboptimal solutions and how it can be encouraged to learn the correct parity function instead.

Questions for authors

Please see above.

3 comments

Thank you for your review. We are pleased you found our paper interesting and the analysis clear and easy to follow. We respond to your points:

As the authors admitted in the conclusion section, the theoretical setting is too simplified. The theoretical analysis considers an idealized STaR objective with linear autoregressive models, while the experiments use GRPO with Transformers. Moreover, the theory assumes optimal policy updates without analyzing the actual optimization process.

We believe that our analysis takes a necessary first step. As prior results on reinforcement learning of autoregressive networks are scarce, we believe it is natural to first understand how RL changes the dynamics in an architecture-agnostic way and then attempt to obtain a finer-grained understanding. As we showed in our analysis, this already offers a mathematically rich landscape with unique phenomena worth understanding (e.g., the form of the shifted distributions over the RL rounds).

Furthermore, in response to your question and that of reviewer 34vr, we added a section to the appendix (Appendix I) and a reference to it in the main text, where we demonstrate that it is possible to extend the analysis to cover the GRPO objective. Assuming the same linear autoregressive policy, we show that one population GRPO update introduces agreement-dependent behavior at the first output position. This implies that the GRPO objective similarly encourages the formation of the low-dimensional feature gkg_k and could similarly lead to shortcut solutions. While we did not want to rush to include a full end-to-end result for GRPO, we believe that Appendix I, and Proposition 7 in particular, show that this might in fact be possible. We also explain there a promising way to achieve this (i.e., by considering the limit points of gradient flow, which is quite different from the STaR analysis). It would indeed be interesting for future work to study this and perhaps obtain a fine-grained understanding of the duration of the suboptimal plateaus observed in the experiments, and whether these depend more strongly on the loss. Understanding optimization dynamics of RL with transformers remains an important outstanding problem.

The interpretation of acquiring new capabilities may be overstated. In my view, learning a low-dimensional feature to identify input regions where the pre-training and post-training targets agree does not necessarily constitute a meaningful new reasoning capability. Instead, it can be viewed as an undesirable shortcut that achieves partial accuracy without genuinely learning the new parity function. While understanding this shortcut is interesting, I believe it would be more important to investigate why RL converges to such suboptimal solutions and how it can be encouraged to learn the correct parity function instead.

Thank you for the suggestion. Following your suggestion, we experimented with a few different strategies that could prevent the model from converging to a suboptimal strategy, and we found that exploration, in the form of increased temperature, can help the model reach strategies that achieve perfect generalization and consistently output "faithful" reasoning. In particular, in Figure 11 (Appendix B), we show that increasing the RL sampling temperature leads to many more random seeds converging to perfect accuracy, often doubling the chances of this happening compared to the temperature 1.0 baseline. We believe this provides additional evidence supporting similar remedies that have been proposed in the literature on frontier LLMs. Please consider taking a look at the paragraph titled "Exploration helps prevent shortcuts" in Appendix B, and let us know if you have any questions.

I appreciate the authors' efforts to address my concerns.

The new GRPO analysis in Appendix I and the additional experiments on increased sampling temperature provide useful insights and partially address my concerns. While the theoretical analysis still relies on a simplified setting and does not fully capture the optimization dynamics of Transformers, I understand that developing a more general theoretical characterization is challenging and may be beyond the scope of the rebuttal period.

The observation that increased exploration can help avoid suboptimal shortcut strategies makes sense to me. Intuitively, exploration is a useful method to escape local minima. Also,maybe modifying the post-training data, for example by introducing greater diversity in CoT formats or reasoning traces, could reduce the tendency to learn such shortcuts in the first place. There has been related work exploring the role of data diversity in other tasks and settings. This might offer a more data-centric perspective on the problem. I understand that this would likely constitute a separate research question, and I do not ask the authors to investigate it as part of the current work. I simply share this thought here.

Overall, I appreciate the authors' thoughtful response and the additional efforts made in the revision. I remain supportive of accepting this paper.

Thank you for your reply and for sharing your thoughts on exploration.

Reviewer_34vrOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates whether RL can improve performance on tasks that were not explicitly learned during pretraining, using the classical problem of learning parity functions over uniformly distributed input bits as a testbed. The authors develop an effective theoretical framework for RL dynamics that characterizes certain suboptimal post-training policies and links them to the formation of low-dimensional features. They find that these features can be learned efficiently but support predictions for new targets only over a subset of the input space. Furthermore, the paper shows that the RL objective transforms target shift into covariate shift, thereby encouraging the model to discover these features. This enables the model to reuse chain-of-thought reasoning acquired during pretraining on inputs for which the original and new targets produce consistent outputs.

Critical strengths
  1. The paper turns an ambiguous question into an interpretable framework. By showing how target shift is transformed into covariate shift, it provides a concrete characterization of the model’s learning dynamics.

  2. Under the same pretrained model and with the same amount of information, SFT performs at near-random levels, whereas RL exhibits increasing response lengths, reuses pretrained cot reasoning, and selectively gates inputs. These findings provide a compelling explanation for how RL can enable models to learn new tasks.

Critical weaknesses
  1. The experiments use GRPO, whereas the theoretical analysis focuses on the star objective and assumes a position-wise linear autoregressive policy rather than a Transformer.

  2. The SFT baseline uses only short-answer supervision, which is a known challenging setting and therefore represents a relatively weak baseline choice.

Questions for authors
  1. can the authors investigate whether one or multiple rounds of RFT can reproduce the gains observed with GRPO? This could further clarify the mechanisms underlying the improvements brought by RL.

  2. can authors extend their theoretical analysis to GRPO to better explain its learning dynamics and performance gains?

3 comments

Thank you for your review, and we appreciate your time and effort. We respond to your points and questions:

The experiments use GRPO, whereas the theoretical analysis focuses on the star objective and assumes a position-wise linear autoregressive policy rather than a Transformer. can the authors investigate whether one or multiple rounds of RFT can reproduce the gains observed with GRPO? This could further clarify the mechanisms underlying the improvements brought by RL.

First, let us defend our analysis by arguing that prior work has also resorted to analyzing linear autoregressive policies to understand some properties of autoregressive learning that are not specific to transformers (e.g., [1, 2, 3]). Our contribution here is to uncover a mechanism that leads to "shortcut" solutions favored by RL, based on the the emergence of low-dimensional features. This mechanism is not a feature of the transformer architecture, and also arises in simpler linear autoregressive networks.

Regarding the question about experiments with STaR and rejection-based fine-tuning, we kindly point you to Appendix B and Figure 8, where we have included such results as part of the original submission. We observed results qualitatively similar to those of GRPO, with the presence of suboptimal solutions as well as runs that succeed in generalizing perfectly. These experiments use 200 RFT rounds with 20,000 samples per round. Please see Appendix C for experimental details. The results support the analysis of the STaR objective, as the shortcuts are not a particular feature of the GRPO method. We added an explicit pointer to these results in the main text.

The SFT baseline uses only short-answer supervision, which is a known challenging setting and therefore represents a relatively weak baseline choice.

Please also see our response to Reviewer Nkv9, where we addressed a similar point.

In short, we deliberately chose to consider methods that only have access to input-output pairs from the new parity function, without CoT supervision readily available, as is often the case in practice. To clear up any ambiguities, we included the SFT with CoT experiments in Appendix B and confirmed, as expected from prior work [1, 4, 5], that the pre-trained transformer is able to perfectly learn the new target, without any plateaus at suboptimal solutions.

can authors extend their theoretical analysis to GRPO to better explain its learning dynamics and performance gains?

Thank you for the suggestion. We added a section to the appendix (Appendix I), where we demonstrate that it is possible to extend the analysis to cover the GRPO objective. Assuming the same linear autoregressive policy, we show that one population GRPO update introduces agreement-dependent behavior at the first output position. This theoretically implies that the GRPO objective similarly encourages the formation of the low-dimensional feature gkg_k and could similarly lead to shortcut solutions. While we did not want to rush to include a full end-to-end result for GRPO, we believe that Appendix I, and Proposition 7 in particular, show that this might in fact be possible. We also explain there a promising way to achieve this (i.e., by considering the limit points of gradient flow, which is quite different from the STaR analysis). It would indeed be interesting for future work to study this and perhaps obtain a fine-grained understanding of the duration of the suboptimal plateaus observed in the experiments. Please consider taking a look at Appendix I, and let us know if you have any questions.

References

  1. Malach, E. Auto-Regressive Next-Token Predictors are Universal Learners.
  2. Tsilivis, N., Malach, E., Ullrich, K., and Kempe, J. How Reinforcement Learning After Next-Token Prediction Facilitates Learning.
  3. Chen, F., Huang, A., Golowich, N., Malladi, S., Block, A., Ash, J. T., Krishnamurthy, A., and Foster, D. J. The Coverage Principle: How Pre-Training Enables Post-Training.
  4. Joshi, N., et al. A Theory of Learning with Autoregressive Chain of Thought.
  5. Kim, J., and Suzuki, T. Transformers Provably Solve Parity Efficiently with Chain of Thought.

Thanks for your details experiment, i will keep my postive scores

We are glad your concerns are resolved.

Reviewer_Nkv9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors demonstrate that RL successfully adapts to a shifted target task, whereas SFT completely fails. The theoretical and empirical analysis (via parity functions) reveals that RL achieves this by converting a target shift into a covariate shift, incentivizing the model to exploit a low-dimensional shortcut to reuse pre-trained CoT computations. Ultimately, the study highlights a critical vulnerability in RL post-training: the model's generated reasoning can be entirely unfaithful, acting as a form of reward hacking that often traps performance at a suboptimal accuracy plateau.

Critical strengths
  1. The paper's rigorous experimental design uses parity functions to mathematically prevent SFT from finding shortcuts, perfectly isolating the true impact of RL.
Critical weaknesses
  1. Missing Crucial Baseline. The experiments lack an "SFT with CoT" baseline, leaving it unclear whether SFT's failure is due to its objective function or merely because it was deprived of intermediate reasoning steps.

  2. Lack of Representativeness. Training a small GPT-2 model from scratch on synthetic data is too simplistic to reliably represent the complex behaviors and generalization capabilities of modern large language models.

  3. Absence of Solutions. The study merely demonstrates a specific instance of reward hacking without proposing any actionable mitigation strategies to solve the resulting problem of unfaithful reasoning.

Questions for authors

See the above weaknesses, especially 1 because the paper is incomplete without 1.

7 comments

Thank you for your review, and we appreciate your time and effort. We respond to your concerns:

Missing Crucial Baseline. The experiments lack an "SFT with CoT" baseline, leaving it unclear whether SFT's failure is due to its objective function or merely because it was deprived of intermediate reasoning steps.

See the above weaknesses, especially 1 because the paper is incomplete without 1.

As mentioned in the introduction (lines 090-091), "We assume access only to input–label pairs from this new parity, with no CoT demonstrations". That is, we intentionally place ourselves in the challenging setting where we do not have access to CoT traces from the task of interest, as we believe this is a situation that can often arise in practice. We therefore compare RL and SFT under the same supervision constraints, with both methods having access only to input-label pairs from the new task. Note that fine-tuning the pre-trained model with CoT traces from the new parity uses much more supervision than either of the methods we study. In fact, as mentioned in the introduction (lines 060-061), parity functions are learnable through next-token prediction with suitable intermediate supervision [1, 2, 3]; hence, we expect the pre-trained model to be able to perfectly learn the new target from CoT traces. To clear up any confusion about this, we performed SFT with CoT traces for the new parity and included the results in Appendix B of the revised PDF (paragraph titled: "SFT with CoT" and Figure 10). We confirmed that the model is able to learn the new target function without any plateaus at suboptimal strategies.

Lack of Representativeness. Training a small GPT-2 model from scratch on synthetic data is too simplistic to reliably represent the complex behaviors and generalization capabilities of modern large language models.

Partially, we agree with you, and we do not claim that our study can represent all the phenomena and capabilities that arise during RL of frontier-scale systems. However, we believe our study on synthetic data is valuable as it allows us to:

  1. Isolate some of the learning phenomena that people encounter as they train modern large language models (sudden phase changes and plateaus at suboptimal accuracy).
  2. Model them using canonical settings from learning theory and computer science. In particular, parity tasks are known to pose computational barriers to learning, despite their simple definition, and they have become widely used as a benchmarks for evaluating learning algorithms, including LLMs (e.g., [4,5,6]).
  3. Rigorously describe complex behaviors, such as “learning new skills,” “unfaithful reasoning,” etc.
  4. Provide mathematical insights into the nature of these behaviors, such as our finding on the formation of low-dimensional features that help with learning “shortcuts”.

Absence of Solutions. The study merely demonstrates a specific instance of reward hacking without proposing any actionable mitigation strategies to solve the resulting problem of unfaithful reasoning.

Thank you for the suggestion. We experimented with a few different strategies that could prevent the model from converging to a suboptimal strategy, and we found that exploration, in the form of increased temperature, can help the model reach strategies that achieve perfect generalization and consistently output "faithful" reasoning. In particular, in the revision we added Figure 11 (Appendix B), which shows that increasing the RL sampling temperature leads to many more random seeds converging to perfect accuracy, often doubling the chances of this happening compared to the temperature 1.0 baseline. We believe that this provides additional evidence supporting similar remedies that have been proposed in the literature on frontier LLMs. Please consider taking a look at the paragraph titled "Exploration helps prevent shortcuts" in Appendix B, and let us know if you have any questions.

We would greatly appreciate it if you could let us know whether your concerns have been resolved and whether you would reconsider our work as worthy of acceptance.

References

  1. Malach, E. Auto-Regressive Next-Token Predictors are Universal Learners.
  2. Joshi, N., et al. A Theory of Learning with Autoregressive Chain of Thought.
  3. Kim, J., and Suzuki, T. *Transformers Provably Solve Parity Efficiently with Chain of Thought.
  4. Barak, B., et al. Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limit. NeurIPS, 2022.
  5. Anil, C., et al. Exploring Length Generalization in Large Language Models. NeurIPS, 2022.
  6. Zhou, H., et al. What Algorithms Can Transformers Learn? A Study in Length Generalization. ICLR, 2024.

Dear Reviewer,

As the author-reviewer discussion period is ending very soon, could you please review and respond to the rebuttal?

Thank you.

Disclosure: I have read the rebuttal myself and drafted a response. The below is the AI refined response based on the draft.


Thank you for the response. However, my core concerns remain unresolved, and I will maintain my score:

Baseline Fairness: Comparing RL (with extended CoT rollouts) against short-answer SFT remains unequal in test-time compute. Under the same supervision constraint, you can create a fair baseline using rejection sampling (e.g., STaR) on the model's own generated CoTs filtered by outcome labels.

Lack of Representativeness: Framing this as a learning-theory study does not show whether low-dimensional "agreement gates" persist in modern pre-trained LLMs with rich semantic priors, rather than being an artifact of orthogonal Boolean parity tasks on a 4-layer scratch-trained GPT-2.

Inadequate Mitigation: Increasing sampling temperature (Figure 11) merely amplifies random exploration and last-token noise in a low-dimensional synthetic setup; many seeds still fail, and stochastic noise injection is not a scalable solution for reward hacking in complex reasoning tasks.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.