acceptodds
Withdrawn from ICLR 2027

Understanding Training Stability in LLM Reinforcement Learning: A Survey

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning has become a key paradigm for enhancing capabilities of large language models, enabling remarkable advances in mathematical reasoning, code generation, and long-horizon tasks. However, compared with supervised fine-tuning, RL optimization is substantially more unstable, frequently suffering from loss spikes and policy collapse. Despite extensive surveys on RL algorithms and alignment, training stability in LLM RL has not been systematically reviewed. This paper provides a unified survey of stabilization techniques from an optimization pipeline perspective. We categorize existing methods into two stages: gradient generation, covering data sampling, importance sampling, and credit assignment; and gradient propagation, covering gradient regulation and numerical/system alignment. Based on this unified perspective, we systematically review representative methods, analyze their underlying stabilization mechanisms, summarize their advantages and limitations, and identify open challenges and promising directions toward stable and scalable RL for LLMs. We hope this survey serves as both a comprehensive reference for future research.

Primary area:
reinforcement learning
Keywords:
reinforcement learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TiF1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This survey groups methods for stabilizing LLM reinforcement learning by where they act in training. It separates gradient generation, covering sampling, importance ratios, and credit assignment, from gradient propagation, covering gradient regulation and numerical consistency. It reviews representative methods, their limitations, and directions for future work.

Critical strengths
  • Organizing methods by their intervention point makes a broad literature easier to navigate. Figure 1 and Appendix Table 1 provide a useful overview. Appendix A.1 also acknowledges that a method can belong to more than one category.

  • Section 4.2 covers precision, quantization, operator differences, and hardware faults alongside algorithmic causes of instability. Its distinction between computing an update reliably and choosing a good update is especially useful for systems with separate training and rollout engines.

Critical weaknesses
  1. The survey covers too much ground at too little depth. Sections 3–4 mostly introduce methods and their intended benefits. There is some useful discussion of limitations, but too little comparison of when methods work, where their findings disagree, or how they interact. For example, Section 3.2 groups prefix-, sequence-, and staleness-aware ratio methods without enough guidance on their respective use cases and tradeoffs. The taxonomy helps organize the literature, but I do not yet see enough synthesis beyond a well-organized reading guide.

  2. The evidence is not compared at the level of individual methods. The appendix tables summarize broad categories, with little detail on training settings, stability measures, or reported outcomes. This makes it hard to distinguish demonstrated benefits from plausible explanations. A focused comparison of representative studies would be more useful than adding more methods. The literature selection criteria and cutoff date should also be stated.

Questions for authors

See weaknesses.

Reviewer_P297OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This survey examines training stability in reinforcement learning with verifiable rewards (RLVR) from a gradient perspective. It organizes the literature around two main themes: gradient generation and gradient propagation, each divided into three subcategories. The authors provide a comprehensive overview of relevant studies within each subcategory.

Critical strengths
  1. The gradient perspective on RLVR training stability is novel and insightful. Figure 1 is particularly effective in illustrating this perspective.
Critical weaknesses
  1. I have reservations about the paper’s suitability for ICLR in its current form. Given its focus on synthesizing existing literature, it may be better suited to a journal or a dedicated survey track.
Questions for authors

NA.

Reviewer_1yUjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This survey reviews stabilization techniques for RL training of LLMs. Building on the gradient of the importance-weighted policy-gradient objective, it groups methods into two stages: gradient generation (data sampling, importance-sampling ratio design, credit assignment) and gradient propagation (gradient regulation, numerical/system alignment). It then briefly describes representative methods in each category. There are no experiments or new theory; the main contribution is the taxonomy.

Critical strengths
  1. End-to-end perspective. The survey covers the full pipeline, from data sampling to numerics and hardware. It brings system-level causes of instability (e.g., training–inference mismatch, non-deterministic kernels) together with algorithmic methods. These causes matter in practice but are rarely covered in existing surveys.
  2. Timely topic. Training stability is a real bottleneck in large-scale LLM RL, and no dedicated survey on it exists yet.
Critical weaknesses
  1. The taxonomy is inconsistent. The paper switches between two classification criteria: where instability originates and where a method intervenes. Some placements are therefore questionable. For example, fixes for training–inference mismatch change the old policy's log-probabilities, which affects gradient generation, yet they are classified under gradient propagation. GSPO also appears in two categories. Because the taxonomy is the main contribution, this undermines the paper's central claim of a unified framework.
  2. Limited depth and coverage. The paper never defines instability or distinguishes its failure modes. The gradient-propagation stage relies heavily on pre-training or general deep-learning work. Meanwhile, several central LLM-RL lines are missing or barely discussed: entropy-collapse control, loss-aggregation bias, truncated importance sampling for training–inference mismatch, and critic stability.
Questions for authors
  1. It would be helpful to state a single classification criterion explicitly and to explain how it applies to borderline cases, such as rollout-side numerical fixes and GSPO.
  2. Some clarification on the scope of the survey would be useful: specifically, the rationale for including pre-training or general-purpose methods, and whether core LLM-RL stability work (e.g., entropy control, truncated importance sampling, critic stability) could be incorporated.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.