acceptodds
Under review as a conference paper at ICLR 2027

Learning Before Reward: Unrewarded Exploration in Large Language Models

Abstract

Recent advances in the reasoning capabilities of large language models (LLMs) have been driven largely by reinforcement learning with verifiable rewards (RLVR). However, it remains unclear whether LLMs can improve through exploration even in the absence of rewards (i.e., neither external verifier rewards nor internally constructed rewards), and whether such exploration can benefit subsequent optimization once rewards are introduced. Blodgett's classic maze experiments showed that rats could acquire knowledge of their environment even before receiving rewards. Motivated by this finding, we study a training setting in which LLMs first undergo training without rewards, called unrewarded exploration, where LLMs learn from their own generated responses without any reward signals, and are then further optimized once rewards are introduced. Across several model families and task domains, we find that LLMs achieve measurable performance gains from the unrewarded exploration phase, and that these gains translate into further improvement once rewards are introduced. In addition, models trained under this setting ultimately outperform counterparts trained with standard RL throughout the training process. To explain these empirical findings, we provide theoretical analyses showing why unrewarded exploration can yield performance gains, shedding light on these dynamics. Taken together, our results suggest that LLMs may benefit from an initial phase of unrewarded exploration, with performance gains becoming more pronounced when rewards are introduced.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, reinforcement learning, reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_MgAzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates a two-stage post-training procedure motivated by unrewarded learning. In the first stage, models train on their own sampled responses using a GRPO-style objective with every advantage fixed to one, without response-dependent reward assignment or correctness filtering. Standard rewarded training follows. Experiments across Qwen and Llama models on mathematical reasoning and GUI tasks report gains during the first stage and in several settings improved performance after subsequent RL. The paper also presents theoretical analyses intended to explain improvement under constant-advantage updates.

Critical strengths

The study examines both the initial reward-independent phase and its effect on subsequent rewarded training, rather than reporting only an intermediate improvement. It covers multiple model sizes and two task domains, and the GUI comparison explicitly matches total training steps between the two-stage procedure and rewarded training throughout.

Critical weaknesses
  1. The contribution beyond a self-training initialization is not isolated.

    The method is essentially a constant-advantage phase that provides a new starting point for standard RL. Its value should be shown against other reward-free initializations, notably SePT (acknowledged in Appendix E.1). The key missing comparison is self-SFT/SePT →\to RL vs. constant-advantage →\to RL under matched budgets; neither the DAPO nor the INTUITOR experiments in Appendix B provide this.

    Also, Tables 1–2 report best-checkpoint results (best average, or per-benchmark maxima), which overstate deployable performance. Given the modest gains, results from fixed or validation-selected checkpoints are needed.

  2. The theoretical analysis is flawed.

    Eq. (23) defines w(a)=π⋆(a)/πref(a)w(a) = \pi^\star(a)/\pi_{\mathrm{ref}}(a), but Eq. (24) uses

    ∑aπ⋆(a) u⋆(a)=∑aπold(a) w(a) u⋆(a),\sum_a \pi^\star(a)\,u^\star(a) = \sum_a \pi_{\mathrm{old}}(a)\,w(a)\,u^\star(a),

    which only holds with πref\pi_{\mathrm{ref}} in place of πold\pi_{\mathrm{old}}. So Eq. (25)'s improvement over the old policy is not established; fixing the measure yields a comparison to the reference policy, not monotonic improvement. Moreover, the assumed alignment between the likelihood ratio and latent utility is itself what needs justifying, and the paper does not explain why constant-advantage training would produce it. Since the theory is a claimed main contribution, this is a central issue.

Questions for authors
  1. Does the first stage outperform unfiltered self-SFT/SePT and internal-reward pretraining when each is followed by the same RL under matched total budgets?
  2. Can you fix the measure inconsistency in Eqs. (23)–(25) and state which policy the guarantee is relative to? How does the corrected result explain the iterative training process?
  3. Were the checkpoints in Tables 1–2 selected on the same benchmark scores that are reported? What are the results with a fixed or validation-selected checkpoint with variability across runs?
1 comment

Thank you for the response and additional experiments. The validation-selected results address my checkpoint concern, and the theorem now correctly states its guarantee relative to the reference policy. However, my main concerns remain:

  1. In the new comparison, the 1.3-point margin over INTUITOR → Rewarded comes entirely from AIME24 and AMC23 (about three problems). On the other four benchmarks the two methods are tied. This is a single run on one model.
  2. Self-SFT and INTUITOR show similar first-stage gains and reach similar levels after RL. The effect therefore looks general to training on self-generated data, not specific to constant-advantage training.
  3. The successive-update analysis shows when improvement occurs, but not why constant-advantage training would produce the required alignment, and it applies to an idealized optimizer rather than the GRPO update actually used.

I will therefore keep my rating of 2.

Reviewer_KJmCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The question this paper is trying to address it to see if training by having more exploration without reward can help. The paper argues that LLMs do achieve gains without reward (unrewarded exploration) and further improves when reward is introduced. The paper has experimented on various types of tasks, such as math reasoning and GUI tasks, multiple types of models, such as qwen and llama. The experimental results show that introducing unrewarded exploration improves quality, more so than ones that do rewarded exploration throughout.

Critical strengths

This paper is well written and the problem statement is clear. The experiments are also pretty comprehensive and provides code, along with full hyperparameter. The experiments also include multiple model families, different task types and multiple datasets.

Critical weaknesses

How many steps are used for "unrewarded exploration"? Seems like the number of unrewarded step is varying based on Appendix C? Figure 2 seems a bit interesting... There is a big difference between Llama and Qwen on the unrewarded exploration phase. Why is that? Looking at D2, I don't see a clear pattern of improvements. How can we know this is not just a randomness artifact?

Questions for authors

See weaknesses.

Reviewer_X1CuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a RL algorithm which consist of two stage: 1) unrewarded exploration where all advantages are set to 1 and 2) standard RL optimization. The experiments show that on Mathematical Reasoning Tasks and GUI Agent Task, the proposed approach consistently outperform the designed controlled baselines.

Critical strengths
  1. The proposed approach is simple.
  2. The paper is clear and easy to follow.
  3. The authors evaluate the proposed approach on two model family with different capacity.
Critical weaknesses
  1. The selected foundation models are too weak and old. For example, the paper uses LLaMa 3.1 and Qwen2.5 series, which were published more than one years ago. I suggest the authors can add more experiments on more latest model and reasoning models.

  2. The baseline is not limited. I think the unrewarded stage seems to be similar to self-training, where model self-generate training data without any filtering. I hope the authors can clarify the difference and add some comparison between these two kind of approaches.

Questions for authors

See above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.