acceptodds
Withdrawn from ICLR 2027

Gen-rl: Rewarding Reasoning That Transfers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In standard reinforcement learning with verifiable rewards (RLVR), a reasoning chain is rewarded for solving the particular problem that elicited it. But the reasoning we ultimately want models to learn should transfer: a broadly reusable strategy and an idiosyncratic solution should not be equally valuable. We introduce generalization-rl (*gen-rl*), which uses in-context learning as a direct signal for transfer. The model first produces a response to one problem, which is left in context as a demonstration for a set of subsequent problems. The first response is rewarded according to how well it enables downstream problem solving via in-context learning, while the downstream responses are themselves trained to make effective use of the demonstration and produce correct answers. We derive a principled policy-gradient estimator for learning this behavior end-to-end. We study transfer in one concrete form, easy-to-hard generalization: whether reasoning learned on problems of one difficulty extends to harder ones. Across multi-digit multiplication, knights & knaves, and standard competition math benchmarks, gen-rl outperforms standard RLVR both on problems as difficult as the training problems and on harder ones from held-out difficulty tiers. Gen-rl's improvement over standard RLVR is largest on the harder tiers, suggesting that it promotes reasoning strategies that extrapolate beyond the training distribution.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning, Reinforcement Learning with Verifiable Rewards, RLVR, Large Language Models, Reasoning, In-Context Learning, Easy-To-Hard Generalization, Generalization

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_UQPoOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a post-training framework that uses in-context learning as a training signal to encourage transferable reasoning. The model first generates a response to an initial question, and this question–response pair serves as an in-context demonstration for a set of subsequent probe questions. The initial response is rewarded according to the model’s performance on these probes.

Critical strengths

Using reinforcement learning to improve the usefulness of model-generated in-context demonstrations is a valuable research direction.

Critical weaknesses
  • The contribution of the initial response is not sufficiently established. Beyond the intuitive explanation based on in-context learning, the paper does not clearly demonstrate why or how much the initial response helps solve subsequent questions. Concrete examples and direct empirical evidence would strengthen the proposed motivation.
  • The experimental evaluation is limited in scope. The experiments rely substantially on toy reasoning tasks, and model coverage is narrow, with only small models (<2B, non-thinking).
Questions for authors
  • The actual teaching effect of the first response remains insufficiently demonstrated. Could the authors directly evaluate the teaching effect of the initial response by comparing performance and model behavior with and without a Gen-RL-style prefix, as well as across different candidate prefixes?
  • What are the main advantages of the proposed framework over approaches that extract reusable skills or consolidate knowledge into a knowledge base?
  • The compute-matched (G=72) control generally performs worse than the (G=8) control. Could the authors explain this counterintuitive result? Could insufficient hyperparameter tuning of the (G=72) baseline contribute to the performance gap?
Reviewer_QgezOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Gen-RL, a reinforcement learning method that encourages models to learn reasoning strategies that generalize to harder problems. Instead of rewarding a reasoning trace based on whether it solves the original question, Gen-RL rewards it based on how well it helps the model solve a set of subsequent probe questions through in-context learning. The probe questions are sampled from harder difficulty tiers, and the model is trained on both generating useful demonstrations and correctly solving the probes. The authors derive a policy-gradient objective with group-relative advantages for both types of responses. Experiments on multi-digit multiplication, Knights & Knaves, and competition mathematics show that Gen-RL consistently outperforms standard RLVR, with larger improvements on harder, out-of-distribution problems. The gains also extend to external mathematics benchmarks.

Critical strengths
  1. Interesting and well-written paper, with a clear motivation and a principled mathematical formulation.
  2. The main strength is converting the goal of learning generalizable reasoning into a verifiable RL training objective, without requiring explicit supervision of reasoning strategies.
  3. Good empirical results across three domains, with particularly promising improvements on harder OOD problems and external mathematics benchmarks.
Critical weaknesses
  1. Insufficient controls and baselines. Gen-RL samples probe questions from harder training tiers, so it is unclear whether the improvements come from learning transferable reasoning or simply from training more on harder problems. A curriculum-matched RLVR baseline, using the same Q2 sampling distribution but without A1 demonstrations, would help isolate this. Additionally, the log-reward ablation shows substantial OOD improvements, so a GRPO variant with a similar group-level log reward, but without demonstrations, would help disentangle the contribution of the reward function from the proposed mechanism.

  2. Unclear transfer mechanism for competition mathematics. Since Q1 and Q2 are sampled based on difficulty rather than topic, it is unclear why a demonstration from one topic (e.g., geometry) would help solve a harder problem from another (e.g., algebra). It would be useful if the authors could provide some intuition for why transfer should occur between such unrelated problems.

  3. Questions about the compute-matched baseline. The G=72 GRPO control performs considerably worse than the standard G=8 control. Why is this the case? Since the control matches the number of generations per prompt rather than the diversity and difficulty distribution of training questions, it is unclear whether this provides a sufficiently strong compute-matched comparison.

  4. Limited evidence at larger scales. Experiments are restricted to relatively small models (1.5B–1.7B). It is unclear whether the method would remain effective for larger, stronger reasoning models. Also all runs are single seed, multiple seeds would strengthen the paper.

  5. Missing comparisons. Compute-matched controls are not provided for multiplication and Knights & Knaves. Comparisons with other approaches such as In-Context RLVR and a student-only ablation in which A1 remains in context but receives no advantage, would also help contextualize the gains.

Questions for authors

See weaknesses

Reviewer_udoLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposed GenRL, an RLVR method to encourage reasoning trace learning from one question that generalizes to others. The method evaluates an answer from the model by using it as context for other questions and then gives the corresponding rewards based on downstream question accuracy. The experiments test some generalization settings on multiplication, knights & knaves, and competition math. The results show that GenRL improves over GRPO across different settings. The author also did several ablation studies to see the reward transformation and what led to the improvement.

Critical strengths

This paper tries to propose a new way other than scaling questions or environments to improve generalization of RL. The idea is novel: using in-context learning to evaluate whether a reasoning trace learned during one question can generalize to other problems.

Critical weaknesses
  1. Figure 1 is terrible. Figure 1(a) is crowded, ambiguous, and too detailed. Some boxes overlap each other. And there are multiple arrows and numerical examples competing for attention. It’s more like AI-generated and not suitable as a teaser figure for the paper. The Figure 1 caption is too long to capture the main focus. And Figure 1 (b) shows the y-axis gain ratio instead of absolute performance improvement, such as accuracy, which is also very confusing.

  2. It’s unfair to say Gen-RL and standard GRPO use the same amount of training data. Although both methods use the same training data, Gen-RL also needs to use additional harder training questions through its probe sampling, which likely significantly increases the total response needed for one training step compared to GRPO. The authors need to clarify this point more explicitly in the paper.

3.. Lack of additional ablation studies. Gen-RL introduces several changes compared with the GRPO baseline, such as demonstration context, probe questions, etc. The ablations in Tables 5 and 8 are not sufficient to ablate the benefit of each component. For example, the paper lacks a GRPO baseline with the same demonstration context for comparison with Gen-RL. So it’s hard to tell which part the improvement comes from.

  1. The benchmark and comparison method is too limited. For competition math, the paper only conducts evaluation in a toy setting without commonly used benchmarks like GSM8K. The improvement on AIME 2024 and 2025 is also very incremental. The paper also lacks comparison with other RLVR methods like DAPO. Overall, these make me concerned about the effectiveness of the proposed method.
Questions for authors

Please address the weaknesses noted above during the rebuttal period, especially Weakness 3 on additional ablation studies and Weakness 4 on comparisons with other RLVR methods across more benchmarks. I understand that the rebuttal period is short, but even preliminary comparisons would be helpful.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.