acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Layerwise Perturbation for Stable Off-Policy LLM Reinforcement Learning

Abstract

Training large language models with reinforcement learning often reuses rollout batches and generates them with a separate inference engine. These practices create policy staleness and training–inference mismatch, which can destabilize importance-weighted updates. We introduce Adaptive Layerwise Perturbation (ALP), which learns a Gaussian perturbation scale for each layer and samples hidden-state perturbations during policy updates. The perturbed training policy forms the numerator of an importance ratio against the unperturbed rollout policy; rollout generation and evaluation remain unchanged. A smoothed-policy analysis relates perturbation to mismatch sensitivity and objective curvature. Across single-turn math and multi-turn tool-integrated reasoning with three model backbones, ALP achieves the highest average accuracy among the evaluated methods. In the five-seed multi-turn comparison, Seq-ALP reaches average accuracy versus for the strongest baseline, Token-MIS. ALP also shows controlled KL divergence, smaller importance-ratio tail excursions than Bypass, and higher multi-turn pass@ at fixed rollout budgets. Location ablations favor perturbing all layers over the tested partial-layer and logits-only variants.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, large language model, off-policy

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wVNsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the unstable importance-sampling (IS) updates in existing RL methods that stem from their off-policy nature, and proposes using structured Gaussian noise to ‘fight’ the random noise introduced by off-policy mismatch.

Critical strengths

The experiments are thorough, and the method is simple to apply and gives consistent gains across a variety of settings, which supports its effectiveness. My main concerns are about the motivation.

Critical weaknesses
  1. The paper does not directly show how much IS instability actually hurts training. How large is the mismatch between inference and training in practice? At what level does it start to hurt performance, and below what level can the model stabilize itself?
  2. The single-turn RL experiments are mainly run on small models (1.5B/4B). On larger models, is this IS instability perhaps negligible for training?
  3. If the problem lies in IS instability, an extreme design would be to replace π_old with stop gradient(π_θ), so that the ratio is stably equal to 1. Would that also work?
  4. For these reasons, I don’t think “fighting noise with structured noise” is a natural choice, and the authors need to discuss the motivation further.
  5. Minor: I did not get the purpose of showing the rollout probability in Figures 1 and 2.
Questions for authors

See weaknesses above.

Reviewer_DRnaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

ALP adds Gaussian hidden-state perturbations with learnable per-layer scales during off-policy LLM RL updates. The perturbed policy forms the numerator of the importance ratio, while rollout generation and evaluation remain unperturbed. The paper provides a simplified smoothing analysis and evaluates single-turn math and multi-turn tool-integrated reasoning across three Qwen backbones, including a five-seed comparison.

Critical strengths
  1. The main multi-turn improvement is supported by repeated training runs. Seq-ALP achieves 50.36 average accuracy versus 48.92 for Token-MIS across five seeds (Table 3). This provides credible evidence of practical gains beyond a favorable single run.

  2. The intervention is simple and leaves deployment unchanged. ALP adds only one trainable scale per layer and requires no perturbation during rollout or evaluation. Its integration into both token- and sequence-level objectives makes it a potentially useful training modification.

Critical weaknesses
  1. The central adaptive contribution is confounded by weight decay. Appendix E.1 shows that weight decay on log-scale parameters can explain the observed scale growth. Consequently, the fixed-versus-adaptive comparison in Table 5 does not establish whether task-driven learning or layer-specific adaptation is necessary. This substantially weakens the claimed methodological contribution.

  2. The evidence does not clearly distinguish mismatch mitigation from general regularization. Experiments do not separately vary policy staleness and engine mismatch, while the theory analyzes an input-averaged, unclipped surrogate rather than the implemented multilayer update. The results support empirical usefulness, but leave the proposed mechanism insufficiently established.

Questions for authors
  1. Do any existing controls separate policy-gradient updates to the scales from weight-decay-induced growth, and do individual layers learn meaningfully different scales?

  2. Is there existing evidence that ALP’s benefit increases with policy staleness or engine mismatch, rather than reflecting general noise regularization?

  3. Future-work suggestion: Compare against a shared learnable scale and a predetermined noise schedule matched to the observed scale growth.

Reviewer_fqkPOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Adaptive Layerwise Perturbation (ALP) to improve the stability of off-policy reinforcement learning for language models. ALP injects Gaussian noise with learnable per-layer scales into hidden states during policy updates, while leaving rollout generation and evaluation unperturbed. The paper provides a simplified smoothing analysis and reports improved performance and stability on single-turn mathematical reasoning and multi-turn tool-integrated reasoning tasks.

Critical strengths

The paper offers an interesting robustness-oriented perspective on stabilizing LLM reinforcement learning. Applying adaptive hidden-state perturbations to reduce sensitivity to policy staleness and training–inference mismatch is a natural and potentially valuable methodological direction. The approach is relatively simple and does not require changing the rollout-generation procedure.

Critical weaknesses

Missing response-length analysis. The paper does not report how generated response lengths evolve during training compared with GRPO and other baselines. Response length is an important diagnostic for reasoning-model training, as it affects both the available reasoning budget and computational cost. Matching maximum generation lengths does not ensure comparable actual token usage. Reporting response-length trajectories and cumulative generated-token counts would help assess whether the performance gains are accompanied by increased generation length or computation.

Insufficient comparison of best-performing checkpoints. The paper does not clearly report a systematic comparison of each method’s best-performing checkpoint. This matters because some baselines exhibit late-stage degradation, and the displayed training curves have different stopping points. Consequently, the reported results do not fully distinguish improvements in attainable performance from improvements in resistance to late-stage collapse. The authors should clarify their checkpoint-selection protocol and report both matched-step results and results from checkpoints selected using a common held-out validation criterion.

Questions for authors

see weakness

Comments on the paper

1 comment

Dear all, The discussion phase will end soon. The authors have provided their responses to your comments. Some of you have started involving discussion, but all of us can take action now. If the provided responses do not address (even part of) your concerns, this is your chance to raise it. Please engage with the authors as soon as possible. We really appreciate your efforts! Thanks, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.