- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This survey groups methods for stabilizing LLM reinforcement learning by where they act in training. It separates gradient generation, covering sampling, importance ratios, and credit assignment, from gradient propagation, covering gradient regulation and numerical consistency. It reviews representative methods, their limitations, and directions for future work.
Critical strengths
-
Organizing methods by their intervention point makes a broad literature easier to navigate. Figure 1 and Appendix Table 1 provide a useful overview. Appendix A.1 also acknowledges that a method can belong to more than one category.
-
Section 4.2 covers precision, quantization, operator differences, and hardware faults alongside algorithmic causes of instability. Its distinction between computing an update reliably and choosing a good update is especially useful for systems with separate training and rollout engines.
Critical weaknesses
-
The survey covers too much ground at too little depth. Sections 3–4 mostly introduce methods and their intended benefits. There is some useful discussion of limitations, but too little comparison of when methods work, where their findings disagree, or how they interact. For example, Section 3.2 groups prefix-, sequence-, and staleness-aware ratio methods without enough guidance on their respective use cases and tradeoffs. The taxonomy helps organize the literature, but I do not yet see enough synthesis beyond a well-organized reading guide.
-
The evidence is not compared at the level of individual methods. The appendix tables summarize broad categories, with little detail on training settings, stability measures, or reported outcomes. This makes it hard to distinguish demonstrated benefits from plausible explanations. A focused comparison of representative studies would be more useful than adding more methods. The literature selection criteria and cutoff date should also be stated.
Questions for authors
See weaknesses.