acceptodds
Desk rejected at ICLR 2027

Diagnosing Collapse in Mixed-Policy RL: The Underlying Mechanism and A Unified Resolution Framework

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Compared to pure on-policy reinforcement learning (RL), mixed-policy RL integrates the exploration of online rollouts with the exploitation of off-policy external traces, effectively surpassing the inherent capability boundaries of the base large language model. Nevertheless, directly applying standard importance sampling to integrate external data induces performance degradation. Although prior methods circumvent this instability by redesigning the surrogate objective, the fundamental cause of the collapse and the unifying theoretical mechanism underlying these disparate solutions remain poorly understood. In this paper, we reveal the fundamental cause of training collapse in mixed-policy RL. Our analysis demonstrates that the instability is not explained by the variance of importance ratios, but driven by severe gradient conflicts induced by low-probability external tokens against the rest of the batch. Motivated by this insight, we propose a unified gradient-suppression framework that reveals existing solutions, despite their different formulations, are fundamentally united by an implicit mechanism that suppresses gradient weights of low-probability tokens. To demonstrate the validity and utility of this framework, we derive a direct probability-aware reweighting method as a concrete instantiation. Extensive evaluations across multiple backbones and benchmarks demonstrate the superior performance of this method, while simultaneously validating the theoretical soundness of our framework and its predictive capacity for method design.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Post-training, Mixed RL, LLM

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BnQqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies why mixed-policy RL collapses when an external correct trace is added to the GRPO rollout group and the previous policy is used as a proxy for the unknown behavior policy. The authors argue that the cause is not importance-ratio variance but a conflict between the gradients of low-probability external tokens and those of the rest of the batch. They show that LUFFY and RL-PLUS can both be read as implicitly suppressing these gradients, and propose a simple method that masks external tokens below a probability threshold. Experiments on four backbones show results competitive with LUFFY and RL-PLUS.

Critical strengths
  • Clear and useful diagnosis. The interval-swap ablation (Table 3) is a clean single-variable test: keeping Vanilla IS on low-probability tokens always collapses training, while replacing it there always stabilizes it.
  • The unified view is predictive. The framework correctly anticipates the outcome of three newly designed ratio variants (Table 4), which goes beyond re-explaining existing methods.
  • The method adds essentially no overhead and is evaluated on four backbones and nine benchmarks, including out-of-domain ones.
Critical weaknesses
  • Relation to prior work is underdeveloped. The gradient bound in Eq. 4 and the idea of down-weighting low-probability tokens overlap considerably with Lopti (Yang et al., ICLR 2026), which is not discussed.
  • The gradient-conflict evidence could be more controlled. The low-probability external tokens all have positive advantage, while the remaining tokens include many negative-advantage rollouts, so part of the negative cosine in Figure 4 may reflect this difference rather than low probability itself.
  • Gains are modest on larger models. On Qwen2.5-Math-7B the method improves over RL-PLUS by only 0.2 points (Table 2), and results are from single runs, which makes the advantage hard to judge.
  • Minor issues. The introduction mentions five backbones, but only four are used in the experiments.
Questions for authors

See weakness. I will raise my score if my concern is addressed.

Reviewer_zuyaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates performance degradation in mixed-policy RL with external expert trajectories. It identifies low-probability external tokens as a source of strong gradients that conflict with the remainder of the batch. The authors introduce an effective gradient factor, ρ(p)=pf′(p)(1−p)\rho(p)=p f'(p)(1-p), to analyze several surrogate objectives and propose probability-aware reweighting. Gradient measurements, interval-swap ablations, and downstream evaluations support the usefulness of suppressing low-probability external-token updates.

Critical strengths

Strengths

  1. The paper studies a practically relevant problem: incorporating correct expert trajectories can nevertheless degrade RL training.

  2. The interval-swap ablation is informative. Retaining suppression in the low-probability interval largely preserves performance, whereas modifying only the high-probability interval does not.

  3. The proposed method is simple, and the improvements on Qwen2.5-Math-1.5B and LLaMA-3.1-8B-Instruct are encouraging.

Critical weaknesses
  1. Insufficient positioning against closely related work.

CHORD, “On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weighting” (paper), explicitly motivates down-weighting low-probability expert tokens to avoid disrupting the existing policy and introduces probability-dependent token weighting. Although CHORD uses an auxiliary SFT objective rather than the same mixed-group objective, it is directly relevant to the central design principle here. The manuscript neither discusses nor evaluates CHORD. Consequently, the novelty of probability-aware suppression is not sufficiently established. The potentially stronger contribution is the gradient-conflict diagnosis, which should be distinguished more clearly from the existing weighting principle.

  1. The causal interpretation exceeds the evidence.

Negative gradient cosine similarity and the short-horizon intervention establish local interference with the remaining-token objective. However, improving this objective is not necessarily equivalent to improving final task performance: acquiring external knowledge may require updates that conflict with existing behavior. The interval-swap results strengthen the argument, but do not establish gradient conflict as the general or primary cause of collapse. The paper should also define “collapse” operationally and distinguish it from comparatively lower performance.

  1. Important controls are missing.

The experiments do not sufficiently separate targeted suppression from a general reduction in external-update strength. Useful controls include uniformly reducing the external-loss coefficient and randomly masking the same fraction of external tokens, with update magnitudes matched where feasible. Another informative ablation would retain the external trajectory in advantage computation while disabling its direct token loss, separating the two channels through which external data influence training.

  1. Statistical robustness is unclear.

Training-seed variability is not reported. While some gains are substantial, the Qwen2.5-Math-7B aggregate improvement over RL-PLUS is only 0.2 points, and the proposed method does not outperform RL-PLUS on Qwen2.5-0.5B. These results warrant more qualified claims about consistent superiority.

Questions for authors
  1. How does the method compare with CHORD under matched data and training budgets?

  2. Does targeted low-probability suppression outperform uniform down-weighting and random masking after controlling for external-update magnitude?

  3. How reliably do the measured conflict statistics predict subsequent task-performance degradation across training settings?

Reviewer_j5zhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the underlying causes of training collapse in mixed-policy reinforcement learning, where on-policy rollouts are combined with external off-policy trajectories. The authors argue that training instability is primarily caused by gradient conflicts from low-probability external tokens rather than high importance sampling ratio variance. Based on this observation, they propose a unified gradient-suppression framework that explains the effectiveness of existing methods such as LUFFY and RL-PLUS. The paper further introduces a simple probability-aware reweighting method that explicitly suppresses gradients from low-probability external tokens. Experiments across Qwen2.5 and LLaMA models demonstrate competitive performance on mathematical reasoning and out-of-domain benchmarks.

Critical strengths
  1. The problem is interesting and well-motivated. Understanding why mixed-policy RL suffers from training collapse is important for improving LLM reasoning capabilities with external data.

  2. The empirical observations are interesting. The analysis of gradient magnitude and direction provides useful insights into why low-probability external tokens may cause optimization instability.

  3. The writing is generally clear and easy to follow.

Critical weaknesses
  1. The proposed probability-aware reweighting method is relatively simple. The main idea is to assign smaller weights to low-probability external tokens, which seems like a straightforward heuristic given the observed gradient conflicts.

  2. The proposed method relies on two manually selected hyperparameters, the probability threshold (c) and the low-probability weight (w_1). From Tables 7–9, the performance appears sensitive to these choices, and the optimal threshold varies across experimental settings. It remains unclear whether the same hyperparameters can consistently work across different models and datasets without extensive tuning. It would be helpful to provide a more systematic analysis of their robustness and joint effects.

  3. The experimental evaluation mainly relies on relatively early model backbones, such as Qwen2.5-Math and LLaMA-3.1-8B. I wonder whether the observed gradient conflicts and the effectiveness of the proposed method still hold for more recent reasoning models with stronger reasoning capabilities and longer context windows. Evaluations on newer models, such as Qwen3, would strengthen the generalizability of the findings.

  4. Although the proposed method achieves competitive results, its improvements over the strongest existing baselines are relatively limited. For example, in Table 2, the average score is 53.8 compared to 53.6 for RL-PLUS, an improvement of only 0.2 points. It is unclear whether such small improvements are statistically significant. More repeated runs or uncertainty estimates would help establish the robustness of the results.

Questions for authors

See weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.