acceptodds
Under review as a conference paper at ICLR 2027

Training Foundation Models with a Single Algorithm

Abstract

Foundation models are typically trained in stages: pretraining, midtraining, supervised fine-tuning (SFT), and reinforcement learning (RL), with changes in both data and learning algorithms. We show that this algorithmic separation is not fundamental. By deriving suitable RL objectives as maximum-likelihood fitting to model-generated target distributions, we place all of these stages under a single likelihood-based training algorithm. This perspective turns the foundation-model training lifecycle into a problem of choosing which data to train on, in what proportions, and when, with conventional staged training as a special case. We instantiate this framework in a unified training system that mixes static data and RL generations within a single trainer. Across controlled and real-world experiments, unified training improves the accuracy–compute tradeoff and task coverage while preserving broader capabilities, and provides a stronger starting point for continual learning. This work lays the groundwork for a unified science of foundation-model training, connecting mathematical principles, training infrastructure, and the development of model capabilities.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Foundation Model, Reinforcement Learning, Pretraining, Midtraining, Hybrid RL, LLMs

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bAgKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a shared likelihood objective for pretraining, midtraining, supervised fine tuning, and selected reinforcement learning objectives. The paper derives likelihood targets for reward weighted generations and implements a trainer that mixes generated responses with static data. The 124 million parameter model tests training across stages on controlled reasoning tasks. A 756.8 million parameter model tests data mixing during mathematical reinforcement learning. The evaluations cover reasoning, coding, knowledge retention, and subsequent adaptation.

Critical strengths

The derivation states the boundary of the common objective clearly. In particular, fitting generations from an earlier policy does not generally produce the gradient of reinforcement learning for the current policy. This distinction makes the proposed training formulation useful for analyzing asynchronous training.

The controlled study accounts for pretraining, midtraining, and generation compute. It also expands the comparison to several staged training budgets. The separate loss and policy regularization comparisons help identify when data timing and mixture composition matter.

Critical weaknesses

The evidence for one algorithm across the full training lifecycle is narrow. The only experiment that mixes data before reinforcement learning uses a 124 million parameter model and one synthetic puzzle family. The larger model completes conventional pretraining, midtraining, and supervised fine tuning before the proposed mixing begins. Appendix C.9 also shows that a tuned, separately normalized supervised loss performs better than the shared loss on Difficulty Levels 5 and 6 in some settings. These results do not establish that the shared loss drives the reported gains. The novelty discussion should also address two omitted, published studies [1] interleave reinforcement learning with targeted fine tuning, while [2] apply reinforcement learning to pretraining text. Neither study is identical to this paper, but both bear directly on its lifecycle claim.

The practical claims about compute and capability retention are not supported by a matched comparison. Appendix B.2 states that the larger model experiments do not match total compute. Appendix D.6 reports different generation procedures and reward baselines for unified and staged training. Available generations lag the trainer by 12 to 44 updates, while the age of consumed examples is not recorded. At a 2% mixture, Table 12 reports HumanEval+ accuracy of 10.98%, versus 15.51% for staged training, and MBPP+ accuracy of 19.00%, versus 25.74%. Mathematical pass@1 is 24.16% for both. The paper therefore demonstrates a retention tradeoff, but does not establish broad capability preservation or a practical compute advantage. [3] provide a relevant omitted analysis of how data from the current policy affects forgetting.

[1] Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions.

[2] Reinforcement Learning on Pre-Training Data.

[3] Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting.

Questions for authors

Can the authors compare unified and staged training for the larger model using the same generation procedure, reward baseline, number of fresh generations, and total model compute? This would determine whether the measured difference comes from data mixing.

What criterion supports the claim of preserved broader capabilities when both coding accuracy measures decline at every reported constant mixing proportion? Please distinguish accuracy retention from the reported gains in coverage and later adaptation.

3 comments

Thank you for the careful reading and for noting the explicit treatment of fixed generators and the compute accounting in the controlled study. We answer each weakness and question in turn below.

W1: one synthetic task: We agree that multi-task learning is an important future direction, and we acknowledge it in our existing discussion section already. However, existing RL-only scaling laws such as [4] only focus on single-task scaling laws (note that their multi-task results still only focus on single-task metrics). We believe that understanding multi-task training within RL training is worth its own research program, so it is beyond the scope of this work and we leave it as a future direction.

W1: 0.76B starts with instruction-tuned model: The 0.76B study starts from an instruction-tuned model by design: it measures what mixing real midtraining data into RL does at a scale where we cannot afford a full sweep of pretraining configurations. Our research questions are about the whole pipeline, from pretraining to RL, and answering them takes many complete runs under controlled compute. That is why the full-pipeline study is at 124M and the real-data study at 0.76B.

W1: Shared vs. separately normalized loss: The goal of the majority of our experiments is to provide scientific understanding instead of claiming that unified training is always better. It is not surprising that the separately normalized loss can sometimes outperform the unified (shared) loss under certain configurations. However, even if the end goal is to pick the best objective in practice, note that the best-performing separately normalized loss requires different hyperparameters in different settings. We still believe the results demonstrate that unified loss should be preferred in practice.

W1 / W2: Omitted related work. Prior work adds supervised data to an RL loss, or turns pretraining text into an RL task. We do the reverse: we turn RL into likelihood fitting, so every stage shares one loss. The revision adds all three studies:

  • ReLIFT [1] alternates RL with fine-tuning on solutions to the hardest questions; in our framework, this is a schedule that switches between two sources.

  • RLPT [2] turns pretraining text into an RL task, the reverse direction from ours.

  • Retaining by Doing [3] attributes RL's lower forgetting to on-policy data; our retention results vary on a different lever, the share of static data mixed into RL.

Q1 / W2: Matching generation, reward baseline, fresh generations, and compute. **All of these are already matched at 124M. We interpret the reviewer’s “the larger model” as the 0.76B model. **

As a summary, the new comparisons don't change the order between unified and staged training: at matched compute, unified keeps a SciQ lead of 11.5–16.0 points and staged keeps a coding pass@1 lead of 3.1–4.1 points at every mixing proportion. Mixing raises SciQ from 50.2% to 67% with more mid-training data, though not strictly monotonic increasing.

Matching compute for the 0.76B model leaves the comparison unchanged: unified keeps its SciQ lead and staged keeps its coding pass@1 lead at every budget. Under the paper's FLOP accounting, the mid-training data that unified training adds costs only 0.3–3.2% more than staged RL updates. The continued-trained results across budget-matched steps are presented in the table below.

MixingModelRL updatesMathCodingARC-CSciQ
–Staged (submission)12,00024.16 (68.56)20.63 (44.40)31.4853.00
0.5%Unified (submission)12,00024.27 (68.32)16.51 (46.38)32.3462.00
Staged, compute-matched12,03924.54 (68.61)19.62 (44.58)32.0850.50
1%Unified (submission)12,00023.86 (68.25)15.49 (44.89)33.5366.10
Staged, compute-matched12,07524.42 (68.28)19.59 (44.40)32.5950.10
2%Unified (submission)12,00023.99 (69.61)15.39 (45.51)34.2267.00
Staged, compute-matched12,15724.65 (68.52)19.50 (45.23)32.0051.20
5%Unified (submission)12,00023.85 (69.68)15.93 (45.17)30.1265.10
Staged, compute-matched12,40025.24 (69.36)19.67 (44.31)31.4053.20
Scores are pass@1 with pass@32 in parentheses.

0.76B training without mixing: the 0% mixing run shares the asynchronous trainer, generation, and reward weighting with the mixed runs. Relative to it, mixing lowers held-out corpus loss and raises SciQ performance from 50.2% to 62%–67%, which isolates the effect of the data within the same implementation.

124M, everything matched: in the existing experiment, staged and unified share the trainer, generator, reward weighting, and number of fresh generations, and both use only data generated at most one update earlier; only the data schedule differs.

W2: Policy lag. In the 0.76B experiments, unified training uses responses from earlier policies, whereas staged RL collects responses from the current policy before optimization. In the 124M study, policy lag is the same for staged and unified training: training examples must be generated by a policy at most one update old. Our theory suggests that the unified training should allow this reasonable level of staleness (12-44 steps as discussed in Appendix D.6 for 0.76B experiments). Despite this lag, unified training with 0% mixing matches staged RL on Math (pass@1 24.75 vs. 24.16; pass@32 68.88 vs. 68.56). In the 124M runs, the trainer uses only data generated at most one update earlier, for both methods, so staleness does not differ between the experiments compared. A controlled lag sweep is future work.

Q2 / W2: The criterion for "preserved broader capabilities". You are right that coding accuracy declines. We will replace "preserves broader capabilities" with the specific outcomes, which form a tradeoff:

  • Accuracy: at 0.5–5% mixing, held-out corpus loss decreases as mixing increases. MMLU (37.1 to 40.8) and SciQ (50.2 to 67) improve. Coding pass@1 ends at or below the pre-RL model. Staged training shows the reverse pattern. We will report this as a measured tradeoff of mixing mid-training text into math RL.

  • Coverage: coding pass@32 is higher than the 0% run at every nonzero proportion, and at the 0.5% budget it is higher than compute-matched staged training (46.4 vs. 44.6).

We believe coverage is a better indicator of capability preservation. One reason the model’s pass@1 capability decreases may be the decreased instruction following capability introduced by the diverse midtraining corpus. Since we see that after the same coding RL, unified checkpoints reach higher coding accuracy and coverage than staged ones, except for MBPP+ pass@1 at 0.5% (Appendix D.7), we see the claim of broader capability preservation as valid. We will revise our wording to distinguish accuracy preservation and coverage preservation in the writing.

[1] Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions

[2] Reinforcement Learning on Pre-Training Data

[3] Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting

[4] The Art of Scaling Reinforcement Learning Compute for LLMs

Reviewer_fCJoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a unified framework that formulates pretraining, midtraining, supervised fine-tuning (SFT), and suitable reinforcement learning (RL) objectives as maximum-likelihood estimation over evolving data distributions. Building on MAX RL, the authors derive likelihood-based formulations for reward-guided learning and combine static training data with model-generated RL samples through a shared weighted-likelihood objective and trainer. The framework supports asynchronous data generation and optimization. The authors evaluate unified training against staged training using controlled reasoning experiments with a 124M-parameter model and real-world data-mixing experiments with a 0.76B-parameter model. The reported results characterize the effects of data composition and scheduling on model performance and capability retention.

Critical strengths

1. Unified training formulation. The paper formulates pretraining, midtraining, SFT, and suitable RL objectives within a common maximum-likelihood framework. By combining static data and reward-weighted model-generated samples through a shared objective, it provides a coherent basis for studying training-stage interactions and data allocation.

2. Systematic analysis of training dynamics. The paper evaluates unified training across controlled reasoning tasks and real-world data mixtures. The analysis characterizes trade-offs associated with data mixing proportions and schedules.

Critical weaknesses

1. Limited empirical scale and generalizability. Experiments are restricted to 124M- and 0.76B-parameter models and mostly consider one RL task at a time. The real-world comparisons are not matched in total compute, making it difficult to disentangle the effects of the unified formulation from data exposure, mixing schedules, and asynchronous training.

2. Limited differentiation from prior work. The central likelihood-based RL formulation builds on MAX RL, while related work has already explored joint SFT–RL training and incorporating RL into pretraining. The paper does not sufficiently isolate the novel theoretical or algorithmic contributions beyond combining these existing ideas within a shared training framework.

Questions for authors

1. Novelty clarification: Beyond combining MAX RL with static-data training under a shared likelihood objective, what specific theoretical or algorithmic contributions distinguish this work from prior approaches to joint SFT–RL training and RL-based pretraining?

2. Experimental attribution: Since the unified and staged implementations differ in compute allocation, asynchronous execution, and reward-baseline handling, can the authors provide controlled comparisons that isolate the contributions of the unified objective and data mixing from these implementation differences?

3. Policy staleness: How sensitive are the reported results to policy staleness in asynchronous training? Can the authors quantify its impact on training stability and performance?

4. Loss formulation robustness: Given that separately normalized RL and supervised losses outperform the unified loss in some settings, what evidence shows that the proposed unified objective provides consistent benefits over appropriately tuned alternative loss formulations?

4 comments

We thank the reviewer for recognizing our unified training formulation and the analysis of training dynamics. We hope to address their questions and concerns below.

Scale and Single RL Task

We agree that multi-task learning is an important future direction, and we acknowledge it in our existing discussion section already. However, existing RL-only scaling laws such as [8] only focus on single-task scaling laws (note that their multi-task results still only focus on single-task metrics). We believe that understanding multi-task training within RL training is worth its own research program, so it is beyond the scope of this work and we leave it as a future direction.

Larger models

Our research questions are about the whole pipeline, from pretraining to RL, and answering them takes many complete runs under controlled compute. That is why the full-pipeline study is at 124M and the real-data study at 0.76B. The revision reports the GPU hours of every experiment, roughly 40000 A100 GPU hours (excluding ideation, iterations, tuning, and failed runs). The detailed breakdown of compute usage is listed below. We note that this is a substantial amount of compute, and we unfortunately lack the compute to experiment with larger-scale models, which might require industry-level compute. That said, we do have confidence that our result should generalize well given the rigorous design of our experiments.

StudyWhatRunsA100 GPU-h
124M (Sec. 5.1)Pretraining and mixed pretraining7344
Midtraining and RL, unified and staged1154,615
Ablations (pretraining replay, loss comparison, GRPO)743,325
0.76B (Sec. 5.2)Pretraining (354.5B tokens) and midtraining215,214
Math RL: constant mixing, schedules, staged reference176,388
Follow-ups: coding RL, WebInstruct, code data, REINFORCE, KL218,833
Total23638,719

Novelty relative to joint SFT–RL and RL on pretraining data

To the best of our knowledge, prior work along this line either adds supervised data to an RL loss, or turns pretraining text into an RL task. We do the reverse: we turn RL into a likelihood-style loss, so every stage shares one loss. Joint SFT–RL methods (PPO-ptx [1], LUFFY [2], UFT [3], SRFT [4], CHORD [5], ReLIFT [6]) combine a policy-gradient loss with supervised, pretraining, or off-policy data during post-training, each with its own loss terms and weights. RLPT [7] applies RL to pretraining text. On the other hand, our approach, Off-policy MaxRL (Section 3.2; Appendix H), converts RL to likelihood maximization on a success-conditioned behavior policy, so pretraining, midtraining, SFT, and RL use one loss. Our approach also encapsulates staged training as a special case. Please also refer to the general response for a detailed explanation of our core contribution.

Isolating data mixing from compute, asynchrony, and the reward baseline.

We thank the reviewer for this suggestion. We have run additional experiments and we report the results below. As a summary, these additional ablations don’t change the comparison order between unified and staged training: at matched compute budgets, unified training retains higher SciQ accuracy, while staged training retains higher coding pass@1. Within the same asynchronous trainer and reward-weighting scheme, mixing mid-training data also improves SciQ accuracy and held-out corpus loss relative to 0% mixing. We report these comparisons below.

  • Compute-matched staged training (0.76B): Matching compute leaves the comparison unchanged: At matched compute, unified achieves higher coding coverage and science-question accuracy, with comparable mathematical performance, while staged retains higher coding accuracy. Under the paper's FLOP accounting, the mid-training data that unified training adds costs only 0.3–3.2% more than staged RL updates. The continued-trained results across budget-matched steps are presented in the table below. (Scores are pass@1 with pass@32 in parentheses)
MixingModelRL updatesMathCodingARC-CSciQ
–Staged (submission)12,00024.16 (68.56)20.63 (44.40)31.4853.00
0.5%Unified (submission)12,00024.27 (68.32)16.51 (46.38)32.3462.00
Staged, compute-matched12,03924.54 (68.61)19.62 (44.58)32.0850.50
1%Unified (submission)12,00023.86 (68.25)15.49 (44.89)33.5366.10
Staged, compute-matched12,07524.42 (68.28)19.59 (44.40)32.5950.10
2%Unified (submission)12,00023.99 (69.61)15.39 (45.51)34.2267.00
Staged, compute-matched12,15724.65 (68.52)19.50 (45.23)32.0051.20
5%Unified (submission)12,00023.85 (69.68)15.93 (45.17)30.1265.10
Staged, compute-matched12,40025.24 (69.36)19.67 (44.31)31.4053.20
  • Asynchrony and reward weighting: The 0% mixing run uses the same asynchronous trainer and reward weighting as the mixed runs (Appendix D.6). Relative to it, mixing 0.5–5% mid-training data lowers held-out corpus loss (from +0.074 to between −0.087 and −0.147 nats/token) and raises SciQ from 50.2 to 62–67, so these effects come from the data, but not the implementation. In the synthetic task, the compute allocation, asynchronous execution, and reward-baseline handling are already matched.

Policy staleness.

In the 124M runs, the trainer uses only data generated at most one update earlier, for both unified and staged training. The 0.76B implementations differ in policy staleness: unified training uses asynchronous generations from earlier policies, whereas staged RL collects responses from the current policy before optimization. Our theory suggests that the unified training should allow a reasonable level of staleness (12-44 steps as discussed in Appendix D.6). Despite this lag, unified training with 0% mixing uses the same asynchronous trainer and reward weighting as the other unified runs and matches staged RL on Math (pass@1 24.75 vs. 24.16; pass@32 68.88 vs. 68.56).

Shared vs. separately normalized losses.

The goal of the majority of our experiments is to provide scientific understanding instead of claiming that unified training is always better. It is not surprising that the separately normalized loss can sometimes outperform the unified (shared) loss under certain configurations. However, even if the end goal is to pick the best objective in practice, note that the best-performing separately normalized loss requires different hyperparameters in different settings. We still believe the results demonstrate that unified loss should be preferred in practice.

[1] Training Language Models to Follow Instructions with Human Feedback

[2] Learning to Reason under Off-Policy Guidance

[3] UFT: Unifying Supervised and Reinforcement Fine-Tuning

[4] SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning

[5] On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting

[6] Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions

[7] Reinforcement Learning on Pre-Training Data

[8] The art of scaling reinforcement learning compute for LLMs.

Thank you for the detailed responses and additional experiments. However, I remain concerned about the generalizability of the proposed approach to larger-scale models. Overall, I appreciate the authors' efforts, but the responses do not sufficiently change my assessment. I will maintain my original rating.

Reviewer_kaGSOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes to place pretraining, midtraining, SFT, and RL in a unified training framework by deriving suitable RL objectives as MLE fitting over evolving mixtures of static and generated data. This work also claims that it lays the groundwork for a unified science of foundation-model training. Empirical study is across controlled and real-world experiments, which try to demonstrate that it improves the accuracy–compute tradeoff and task coverage while preserving broader capabilities, and provides a stronger starting point for continual learning.

Critical strengths

The experiments examine accuracy, compute, retention, data mixing, coverage, and continual learning rather than only final task accuracy.

Critical weaknesses
  1. The theoretical novelty is unclear for me. The static-data objectives are standard MLE, and the RL formulation builds on the MAXRL likelihood interpretation. The paper should distinguish its contribution more precisely from this prior work.
  2. The empirical scope is limited to 124M- and 0.76B-parameter models and mostly single-task RL settings (I mainly read the main part of this manuscript). The authors acknowledge these limitations on p. 9, so the empirical evidence remain insufficient.
  3. The lengthy and dense appendix raises concerns about readability, while the extensive AI assistance disclosed in the AI-use statement makes it difficult to assess the authors' specific technical contributions.
Questions for authors
  1. What is the precise theoretical contribution beyond MAXRL and standard MLE?
  2. How robust are the conclusions for larger models and multitask RL?
  3. How sensitive are the gains to the chosen data mixtures and schedules?
5 comments

We thank the reviewer for their thoughtful feedback and suggestions, they would greatly improve our work! We address their comments below.

What is the precise theoretical contribution beyond MAXRL and standard MLE?

  • We show, for the first time to our knowledge, that pretraining, midtraining, SFT, and RL can all be trained with the same objective and the same trainer, so that the stages differ only in the data they use. This turns the question of how to arrange training stages into a question about data allocation that can be studied directly. The contribution has three parts:
  • Off-policy MaxRL. MaxRL [1] treats on-policy RL as maximum likelihood on the model's own successful samples. We extend this to samples from any fixed policy: the standard likelihood loss on a generator's successful completions fits that generator's success-conditioned distribution, and reduces to MaxRL when the generator is the current model (Section 3.2; Appendix G covers REINFORCE [2] and continuous rewards; Appendix H).
  • One objective for every stage. With off-policy MaxRL, RL generations become one more data source alongside pretraining text and demonstrations. All sources can be mixed in the same batch and trained with the same loss, and generation can run asynchronously from training. Staged training is the special case in which the mixture switches from one source to the next.
  • A science of training as one process. With the objective fixed, the data mixture and its schedule are the only things that change between runs. We use this to measure how they shape accuracy, coverage, retention, and continual learning (Section 5). We did substantial experiments to study the science of unified training and showed how our approach can be beneficial in using the data more effectively, retaining broader capabilities from pretraining, and inducing better generalization capabilities as well as broader coverage under controlled studies.

Multi-task RL

We agree that multi-task learning is an important future direction, and we acknowledge it in our existing discussion section already. However, existing RL-only scaling laws such as [3] only focus on single-task scaling laws (note that their multi-task results still only focus on single-task metrics). We believe that understanding multi-task training within RL training is worth its own research program, so it is beyond the scope of this work, and we leave it as a future direction.

Larger models

Our research questions are about the whole pipeline, from pretraining to RL, and answering them takes many complete runs under controlled compute. That is why the full-pipeline study is at 124M and the real-data study at 0.76B. The revision reports the GPU hours of every experiment, roughly 40000 A100 GPU hours (excluding ideation, iterations, tuning, and failed runs). The detailed breakdown of compute usage is listed below. We note that this is a substantial amount of compute, and we unfortunately lack the compute to experiment with larger-scale models. That said, we do have confidence that our result should generalize well given the rigorous design of our experiments.

StudyWhatRunsA100 GPU-h
124M (Sec. 5.1)Pretraining and mixed pretraining7344
Midtraining and RL, unified and staged1154,615
Ablations (pretraining replay, loss comparison, GRPO)743,325
0.76B (Sec. 5.2)Pretraining (354.5B tokens) and midtraining215,214
Math RL: constant mixing, schedules, staged reference176,388
Follow-ups: coding RL, WebInstruct, code data, REINFORCE, KL218,833
Total23638,719

Sensitivity to mixtures and schedules

We thank the reviewer for pointing this out. If we understand the question correctly, we are asked about the sensitivity to data mixtures and schedules, which we have already presented in Sections 5.2.1–5.2.2 and Appendices D.2–D.3 as one of our core results in studying the science of unified training. For the convenience of the reviewer, we report them again below:

  • Proportion: We run experiments with seven constant proportions, between 0–5%, in our 0.76B model experiments. We observed held-out corpus loss to fall steadily with more mixing (from +0.074 to −0.147 nats/token), while performance on SciQ rises from 50.2 at 0% to 62–67 at 0.5–5%.

  • Schedule: Ending mid-training data mixing earlier improves coding accuracy while retaining part of the reduction in held-out corpus loss. We compared four mixing schedules with the same average target mixing proportion of 0.5%. Constant mixing gives 16.5% coding pass@1. Starting at 2%, gradually reducing the proportion to zero halfway through training, and then continuing with RL alone raises coding pass@1 to 18.9%. This schedule retains some of the corpus-loss improvement achieved by constant mixing.

Readability and AI use.

All the technical, ideation, and experiment designs are our own. AI tools helped with standard usage, including code, data processing, figures, literature search, and polishing writing. We tested the code and checked every result ourselves.

[1] Maximum Likelihood Reinforcement Learning

[2] Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning

[3] The art of scaling reinforcement learning compute for LLMs.

Thank the authors for the clarifications. I would raise the score. However, I still hold the concerns on the scalability of the proposed model.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.