acceptodds
Under review as a conference paper at ICLR 2027

What Pretraining and Midtraining Make Learnable from Rewards

Abstract

A reward can identify a correct answer while leaving the computation needed for new inputs undetermined. We study what pretraining and midtraining contribute to this gap. In sequential state computation and contextual memory, we characterize mechanisms that agree on every training reward yet disagree on held-out answers. Task-independent source observations resolve this ambiguity. We then construct finite sampled-Adam paths from random initialization through source prediction and reward adaptation in the same parameters: prediction acquires execution or retrieval, and rewards learn how to use it. Experiments with pretrained Qwen2.5 checkpoints measure this division of labor through source midtraining. With first-operation source supervision, Sequential models acquire execution and show a strong reward-task-selection advantage over a private-random source control. Memory experiments show retrieval retention and task selection under an independent source intervention. Public tasks separate accuracy at reward entry, subsequent gain and final performance. Together, these results explain how source information becomes computation that rewards can use.

Primary area:
learning theory
Keywords:
Self-supervised learning, Reinforcement learning with verifiable rewards, Compositional generalization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_iEMnOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the prerequisites for efficient RLVR: pretraining or midtraining with the so-called source examples are needed to obtain the operation rules. Furthermore, the paper suggests that the model needs to retrieve information from the proper memory and use it to make a decision. Therefore, the source examples teach retrieval and the rewards teach the decision rule.

Critical strengths
  • This paper studies how pretraining and midtraining influence the follow-up RL training, and provides theoretical characterizations and supporting experiments on how the source observations resolve the ambiguity where identical training rewards are obtained but different test answers are required.
Critical weaknesses
  • The presentation of the paper is poor.
    • Many notions are used before introduction. For instance, it is unclear the meaning of Line 32 “Prediction may supply information absent from rewards, build a computation that uses it, or already teach part of the task.”
    • From the description in the second paragraph alone, it is not perfectly clear to me what is source prediction. In addition, it is also not clear to me how Figure 1 explains this notion.
  • This paper lacks comparison with the existing works like [1] and the novelty of the research needs to be justified.

[1] Yuan, et al. From f(x) and g(x) to f(g(x)): LLMs Learn New Skills in RL by Composing Old Ones. 2026.

Questions for authors
  • Can the authors compare their conclusions with [1]? [1] indicates that the atomic operations are needed before RL training, and RL training enables composition operations of these atomic operations.
Reviewer_uwoaOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies what task-independent source prediction contributes to subsequent learning from terminal rewards. In sequential state computation and contextual memory, it characterizes mechanisms that agree on all permitted training rewards but disagree on held-out answers. It then constructs finite sampled-Adam paths that acquire execution or retrieval from source observations and learn a downstream task rule through rewards, retaining parameters across stages. The contextual construction includes a frozen-query/key comparison. Experiments on pretrained Qwen2.5 models distinguish acquisition, retention, and reward-dependent task selection. Sequential shows a substantial source-structure advantage. The original Memory experiment shows retention without a resolved task-selection advantage, whereas a separate confirmation with additional preparation and stabilization shows positive effects. Public-task experiments distinguish reward-entry accuracy, subsequent improvement, and final performance.

Critical strengths

S1. Precise separation of information and optimization.

Theorems 4.1 and 5.1 characterize what the specified training-reward interface cannot identify, rather than attributing every failure to optimization or insufficient reward samples. The contextual error floor also connects missing source observations to an explicit generalization limitation. This provides an informative account of why source data can matter even when more rewards from the same training support are available.

S2. Constructive learning paths and stage-specific evidence.

Theorems 4.2 and 5.2 connect source acquisition and reward adaptation through finite sampled updates, rather than assuming a ready-made executor or retrieval mechanism. The explicit resource accounting distinguishes two optimizer updates from 2B_R verifier queries. Empirically, the matched A/B reward branches and source probes help distinguish acquiring a computation from learning its task-specific use. Reporting the original Memory null result alongside the independent confirmation is particularly useful for understanding this distinction.

Critical weaknesses

W1. The Sequential acquisition guarantee has a restricted representation-learning interpretation.

Appendices H.3.5 and H.5 choose positive but potentially very small backbone and slow-row learning rates, keeping features close to a reference while the readout learns. This is a legitimate constructive result, but does not establish substantial backbone representation learning under ordinary full-parameter midtraining. Appendix B acknowledges the difference between the theoretical initialization and inherited Qwen weights; nevertheless, the main exposition would be more informative if it clearly separated what is learned in the readout, what is acquired in attention, and which learning-rate restrictions are essential. This limits the explanatory reach of the neural construction; it is not a claim that the stated convergence theorem is false.

W2. The positive Memory result does not isolate the transition from retained retrieval to usable task information.

The original replay experiment preserves retrieval but leaves the primary task-selectivity effect unresolved: 0.67 percentage points, with a 95% interval of [-1.22, 2.57] (Section 6.2). The independent confirmation adds public-projection and retrieval-vector supervision, vector/syntax retention, calibrated sampling, and guarded updates (Appendix C.2). Both routes share these procedures, so the relevant-versus-alternative retrieval contrast is meaningful. However, the experiment does not identify which additions turn the original retained computation into a useful reward-learning signal. The authors disclose this limitation, but it still restricts the empirical explanation to the complete recipe rather than ordinary source prediction followed by standard reward adaptation.

Questions for authors

Q1 — Clarification of W1.

Which parts of the Sequential execution capability are learned in the readout versus the backbone, and which rate-separation conditions are essential to the argument? A compact comparison with the contextual attention-acquisition result would clarify the scope of the two constructions; a new theorem for unrestricted Transformers is not requested.

Q2 — Targeted evidence for W2.

Is there a focused ablation or existing diagnostic identifying what bridges the original Memory null result and the positive confirmation? In particular, can the role of retrieval-vector supervision/retention be separated from sampling calibration and update stabilization? This would materially affect how broadly the empirical mechanism can be interpreted; a full factorial sweep is not requested.

Reviewer_NSsKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies what source prediction contributes to later reward learning. It constructs Sequential and Memory tasks with mechanisms that produce identical training rewards but different test answers. The paper also proves finite sampled-Adam learning paths through source prediction and reward adaptation. Qwen2.5 experiments separately measure computation acquisition, retention, and reward-dependent task selection.

Critical strengths
  1. The paper gives precise information characterizations for its task families. Theorems 4.1 and 5.1 identify reward-equivalent mechanisms and quantify the ambiguity remaining after source observations. The contribution lies in these exact results, rather than the general observation that rewards can leave mechanisms unidentified.

  2. The analysis connects both training stages through finite sampled updates. Unlike simply assuming a useful pretrained representation, the constructions track its acquisition and subsequent reward-dependent use in the same parameters. The paired experiments also distinguish acquisition, retention, and task selection

Critical weaknesses
  1. The presentation obscures the specific theoretical advance and its scope. Prior work already studies prediction followed by RL [1], outcome-reward reasoning [2], and pretrained primitive composition [3]. The paper provides substantial theoretical analysis, but the abstract and introduction do not clearly state what new answers it establishes. Dense notation also makes the task assumptions and the mapping from theoretical quantities to empirical metrics difficult to follow.

  2. The experiments do not fully establish the proposed mechanism. Small pretrained Qwen2.5 models test additional midtraining rather than acquisition from random initialization. The original Memory result is inconclusive, while its positive confirmation adds several unablated components. Public-task gains also remain confounded with entry accuracy. These gaps limit the connection between the theory and broader language-model learning.

References

[1] How Reinforcement Learning After Next-Token Prediction Facilitates Learning. 2025.

[2] Provable Benefits of RLVR over SFT for Reasoning Models: Learning to Backtrack Efficiently. 2026.

[3] RL Post-Training Builds Compositional Reasoning Strategies. ICML 2026 Workshop on Compositional Learning.

Questions for authors
  1. Relative to [1]–[3], what is the main new insight: exact reward-equivalence characterizations, finite sampled-Adam guarantees, or a new learning mechanism? Please state the concrete answers in the introduction.

  2. Which assumptions of Theorems 4.1–5.2 hold in each Section 6 experiment? Please distinguish direct mechanism tests from illustrative comparisons and explain the supported scope beyond the constructed tasks.

  3. Could the Section 4.1 toy example explicitly show two reward-indistinguishable mechanisms, their different test answers, and the resolving source observation? Please also map the main theoretical quantities to their empirical measurements.

  4. How much do the Qwen2.5 results depend on inherited pretraining? Please clarify what the matched-checkpoint controls isolate and summarize the available matched-budget random-initialization comparisons.

  5. Which additions in Appendix C.2 are necessary for the positive Memory result? Please separately ablate vector supervision, retention updates, sampling calibration, and guarded steps.

  6. In Table 2, how can improved reward learnability be distinguished from changed entry accuracy or recovery after source-induced degradation? Entry-matched controls or queries-to-target comparisons would help.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.