acceptodds
Under review as a conference paper at ICLR 2027

Learning Perturbation Robust Policies for LLM Agents with Stable Optimization

Abstract

Reinforcement learning (RL) has become an effective post-training paradigm for long-horizon large language model (LLM) agents. However, we find that the resulting policies can be sensitive to various policy perturbations, such as hidden-state noise, pruning, and quantization. In this work, we study how to improve perturbation robustness during policy optimization. We first introduce the notion of a perturbation robust policy and analyze conditions under which perturbed policy updates preserve stable monotonic improvement. Based on this analysis, we introduce Stable Perturbation-Robust Policy Optimization (SPrPO), which applies adaptive and sensitivity-aware perturbations during RL training. We evaluate SPrPO on ALFWorld and WebShop and conduct systematic experiments across multiple perturbation types and scales, showing improved perturbation robustness while maintaining stable policy optimization.

Primary area:
reinforcement learning
Keywords:
LLM Agents, Perturbation Robustness, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_K4DMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how policy perturbations affect RL-trained LLM agents and aims to make these agents more robust when solving tasks. The authors propose Stable Perturbation-Robust Policy Optimization (SPrPO). SPrPO injects adaptive, sensitivity-aware Gaussian perturbations into the FFN channels during training. Experiments on ALFWorld and WebShop cover several types of policy perturbations, including Gaussian noise, dropout, structured and unstructured pruning, and quantization, and SPrPO generally outperforms the baselines.

Critical strengths
  1. The problem is of practical significance. Deploying large models is a challenge in many real-world scenarios, and compression or changes in numerical precision at deployment time can degrade the performance of RL-trained agents.
  2. Broad evaluation across perturbations. The experiments cover five types of perturbations, each at multiple strengths, and the results show that SPrPO generally outperforms the baselines.
Critical weaknesses
  1. Missing comparison with directly training a smaller model. The paper is motivated by the observation that models are often compressed at deployment time, which degrades task performance. The authors therefore propose this framework to improve robustness. However, I have a question: how would a smaller model, such as Qwen2.5-0.5B-Instruct, perform if it were directly trained and deployed? If such a small model already performs well, would the proposed method still be necessary?

  2. Missing relevant baselines. The paper discusses two closely related works, NPFT and QeRL, but does not compare with either of them.

Questions for authors

1.How does a directly trained smaller model (e.g., Qwen2.5-0.5B-Instruct) perform compared with the compressed Qwen2.5-1.5B-Instruct policies trained with SPrPO?

2.How does SPrPO compare with NPFT and QeRL? If such a comparison is not feasible, could the authors explain why?

3.Under settings such as dropout and structured pruning, SPrPO’s success rates remain very low compared with the clean setting. How do the authors view the practical value of the method in these settings?

4 comments

We thank the reviewer for the practical questions. We address each question and weakness below and will incorporate the clarifications and new results into the revised version.


Q1 / W1: How does a directly trained Qwen2.5-0.5B compare with a compressed Qwen2.5-1.5B policy trained with SPrPO?

When a compact policy is needed for deployment, directly training a smaller model is a natural alternative to compressing a larger RL-trained policy, and we compare the two below.

First, quantization causes relatively less degradation to the policy, and it is a widely used model compression approach in practice. The SPrPO policy with INT6 quantization outperforms the directly trained Qwen2.5-0.5B by a large margin.

ModelRL trainingEstimated memoryALFWorld IDALFWorld OODWebShop
Qwen2.5-0.5B (BF16)Vanilla≈1.0 GB46.7±4.438.1±2.034.3±1.1
Qwen2.5-1.5B (INT6)SPrPO≈1.5 GB91.7±1.590.8±1.655.7±1.6

Second, directly pruning the 1.5B policy to the size of Qwen2.5-0.5B is challenging: it requires removing more than 80% of the FFN channels, which leads to a severe performance collapse. In practice, however, pruning is commonly followed by lightweight recovery training on a small amount of data [1]. Following this pipeline, we pruned 80% of the FFN channels of both 1.5B policies with the criterion of LLM-Pruner, which gives 0.62B parameters, and recovered them by distillation from their own unpruned policies on only 500 trajectories.

ModelRL trainingCompressionParametersALFWorld IDALFWorld OODWebShop
Qwen2.5-0.5BVanillanone0.49B46.7±4.438.1±2.034.3±1.1
Qwen2.5-1.5BVanillaprune + recover0.62B81.2±2.778.6±2.232.7±5.4
Qwen2.5-1.5BSPrPOprune + recover0.62B90.0±2.184.8±1.152.2±4.3

Surprisingly, with this lightweight recovery, the pruned SPrPO policy outperforms the directly trained Qwen2.5-0.5B by a large margin at a comparable size, as well as the pruned Vanilla policy. We will include this comparison as supplementary results in the appendix.

Finally, we want to clarify the scope of this paper. Model compression is an important observation and motivation of our work. Our research focus is whether an RL-trained agent remains reliable when facing unknown perturbations, and model compression is one typical such perturbation in deployment. The practical value of SPrPO therefore extends beyond compression, which we explain in more detail in Q3.


Q2 / W2: How does SPrPO compare with NPFT and QeRL?

Thank you for pointing to these works. We acknowledge that QeRL and NPFT have inspired our work, but SPrPO differs substantially from them in the task setting, the training goal, the perturbations considered, and the design of the injected noise, as shown in the table below.

QeRLNPFTSPrPO (ours)
Training paradigmRLSupervised fine-tuningRL
Task settingSingle-turn reasoningStatic language benchmarksLong-horizon, multi-turn agents
Training goalEfficient RL and quantization-aware training (QAT)Post-Training Robustness (PTR) to quantizationPost-Training Robustness (PTR) to unknown perturbations
Perturbations studiedQuantizationQuantizationGaussian noise, dropout, structured and unstructured pruning, quantization
Injected noiseGaussian noise with a shared variancePerturbations on sensitive weightsTraining-stability-aware: less noise on sensitive channels

Because the task setting and training objectives differ substantially, we compare their noise-design principles within the same agent RL pipeline (Sec. 6.4). Our Multiplicative Gaussian baseline uses sensitivity-independent noise, corresponding to QeRL's equal-variance design, while our SAM-style baseline concentrates perturbations on sensitive channels, adapting NPFT's principle of perturbing sensitive components. We will clarify these connections in the revised version.

Q3: What is the practical value of SPrPO?

Although SPrPO still faces large performance drops under the most severe perturbations, it can offer practical benefits for deploying RL-trained agents, as discussed below.

First, our work studies general policy perturbations, which in real-world deployment are not limited to model compression and also arise from different hardware, numerical precision and random noise. For example, RL frameworks such as verl [2] train the LoRA adapter in FP32 and often merge it into the BF16 base model for serving, where the rounding of the learned update perturbs the trained policy. This is the typical pipeline when a small model is fine-tuned with RL for a specific application, such as online shopping, household assistance or mobile device control, which makes robustness to post-training perturbations particularly valuable in routine deployment.

Second, even when a perturbation severely degrades a policy, a more robust policy is a better starting point for recovery. In Q1 / W1, pruning 80% of the FFN channels causes a large performance drop for both 1.5B policies, but after the same lightweight recovery, the pruned SPrPO policy recovers to a much higher level than the pruned Vanilla policy.


References

[1] Xinyin Ma, Gongfan Fang, and Xinchao Wang. LLM-Pruner: On the Structural Pruning of Large Language Models. In Advances in Neural Information Processing Systems (NeurIPS), 2023.

[2] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A Flexible and Efficient RLHF Framework. In Proceedings of the European Conference on Computer Systems (EuroSys), 2025.

I thank the authors for the response and additional experiments, which address most of my concerns.

Thank you for your reading, and we are pleased to see that most of your concerns have been addressed. We are also very grateful for your insightful and inspiring questions: they led us to the prune-and-recover experiment, whose results were a pleasant surprise to us, and we plan to study this direction further in follow-up work.

If any point remains open, please let us know, and we would greatly appreciate it if you could reconsider your assessment based on the additional experiments. Thank you again for your time and constructive feedback.

Reviewer_w7HCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies the robustness of RL-trained LLM agents to deployment-time policy perturbations: hidden-state noise, channel dropout, structured and unstructured pruning, and quantization. It formalizes the goal as (d, eps, beta)-probabilistic perturbation robustness, extends the TRPO monotonic-improvement analysis to the perturbed setting (Thm 1), derives a sensitivity-based stability condition under Gaussian perturbations (Cor 1), and shows that stable updates admit perturbation-robust limit points (Thm 2). Building on this, SPrPO injects multiplicative Gaussian noise into FFN channels during RL training, with per-channel scales adapted to sensitivity and improvement statistics estimated from the current rollout batch. On ALFWorld and WebShop (Qwen2.5-1.5B-Instruct + GiGPO), SPrPO improves success rates under all five evaluated perturbation types while preserving clean performance.

Critical strengths
  1. A timely and well-motivated problem, backed by a thorough empirical study. To my knowledge this is the first work to examine the perturbation robustness of RL-trained LLM agents under a general perturbation family. The motivating observation is striking -- pruning only 20% of FFN channels nearly collapses WebShop performance -- and the evaluation is unusually systematic: five perturbation types across channel-wise and weight-space granularity, full robustness curves, ID/OOD splits, and reward-landscape visualization. As RL-trained agents move toward real deployment under serving constraints, this opens an evaluation axis that prior agentic-RL work has simply ignored.

  2. A simple, inexpensive method whose gains transfer beyond the perturbations seen in training. SPrPO needs no extra rollouts (~1.7x step time from one additional forward/backward pass, Table 4), and the improvements extend to perturbations never injected during training -- unstructured SparseGPT pruning and RTN quantization over all linear weights (WebShop INT6 retention 92.8% vs. 79.3% for Vanilla; ALFWorld unstructured r=0.7, 58.0 vs. 41.1). Gains under such mismatched perturbations are hard to explain by overfitting to the injected noise distribution, which suggests the sensitivity-aware allocation is capturing something real about flatness.

Critical weaknesses
  1. The evidence base is narrow: one small model, one RL algorithm, two benchmarks. Everything runs on Qwen2.5-1.5B-Instruct with LoRA trained by GiGPO. Sub-2B models are known to be disproportionately sensitive to compression, so the effect sizes here may not extrapolate -- gains could shrink at 7B+, on other families, or under full-parameter training, and the baseline ordering could shift as well. The perturbation mechanism is also algorithm-agnostic in principle, yet nothing is shown for other widely used estimators (GRPO/RLOO/IGPO-style). For a paper whose claims are phrased generally about "RL-trained LLM agents," this is the main scope limitation of the evaluation as it stands.

  2. The training-time perturbation model, the theory, and the deployment-time perturbations do not fully line up. Training sees i.i.d. multiplicative Gaussian noise on FFN intermediate channels only; the evaluated deployment perturbations are deterministic, structured, and weight-dependent (Fisher channel pruning, SparseGPT, RTN over all linear weights including attention). Three specific gaps follow. First, Cor 1 -- the only result tying the stability condition (8) to practice -- covers just the Gaussian proxy, so the "general family of perturbations" framing rests on empirical transfer, not on the analysis. Second, Def 1 and Thm 2 assume the perturbation distribution is supported on the bounded set Xi_delta, while Cor 1 and the algorithm use an unbounded Gaussian; this is never reconciled. Third, condition (8) is a feasibility constraint, not an allocation rule: with alpha=1 the left-hand side sum_i sigma_i^2 S_i collapses to drho^2m_hat^2, independent of per-channel sensitivities, so the inverse-sensitivity rule sigma ~ m_hat * S_hat^(-alpha/2) is motivated by the condition rather than derived from it. I checked the main proof steps and did not spot an outright error; the issue is the distance between what is proved and what the algorithm and claims assume, which seems fixable in revision.

Questions for authors
  1. Is the fragility in Fig 1 specific to RL-trained policies? Please report the base instruct model -- ideally also an SFT baseline on the same trajectories -- under identical perturbations. As written, the claim that the tolerance seen on static tasks "does not carry over to RL-trained agents" is argued by analogy to the compression literature rather than demonstrated; the fragility could simply be a property of 1.5B-scale models under any training paradigm. This bears on the paper's core motivation.

  2. How does SPrPO behave on a larger model (e.g., Qwen2.5-7B/14B) and on a non-Qwen family? Do the robustness gains diminish with scale? Evidence here would substantially strengthen the empirical claims.

  3. What, if anything, in SPrPO is specific to GiGPO? Please clarify what changes are needed to combine the perturbation scheme with GRPO/RLOO/IGPO-style advantage estimation, and ideally show one such experiment. If it is plug-and-play, an explicit control stating so would resolve the question.

  4. On the theory (clarifications, not claimed errors): (a) How do you reconcile the bounded-support requirement on nu in Def 1 and Thm 2 with the unbounded Gaussian in Cor 1 and the algorithm? (b) The robustness notion measures deviation below the policy's own mean perturbed return rather than below clean performance -- a policy with low perturbation-induced variance but a large clean-to-mean gap would count as robust. Is this intended, and can the clean-to-mean gap be bounded too? (c) m_hat is the squared PPO gradient norm, standing in for L_nu(theta'), which is a 1/(1-gamma)-scaled advantage expectation; it also drives the noise schedule over training. How sensitive are the results to this proxy choice?

  5. Why perturb only FFN intermediate channels (equivalently, columns of W_down)? Deployment-time quantization and pruning also touch attention and the gate/up projections. If FFN-only perturbation is empirically better or more stable than perturbing all linear layers, that ablation is worth reporting.

  6. Minor: (a) WebShop baselines show large seed variance (Vanilla +/- 4.39) -- confidence intervals would help the key claims. (b) The closest baseline, NPFT (Wang & Yang, 2025), is cited but never compared; an RL adaptation seems feasible and would isolate what the RL-specific design adds. Relatedly, please clarify the novelty boundary with QeRL (Huang et al., 2026) given the "first to study" framing.

5 comments

We thank the reviewer for the thoughtful questions on scope and on the connection between theory and algorithm. We address each question and weakness below and will incorporate the clarifications and new results into the revised version.


Q1: Is the fragility specific to RL-trained policies?

First, we would like to clarify the comparison in our introduction: the contrast we draw is between static tasks and long-horizon agentic tasks, and the fragility stems from the nature of long-horizon agentic tasks. Static tasks mainly evaluate the language ability of the model, which tends to be relatively tolerant to compression: for example, after 20% of its parameters are removed by structured pruning, an LLM still retains about 90% of its performance [1]. Instead, long-horizon agentic tasks require completing a task through sequential interactions, where "a single wrong action early in a long-horizon task can derail the whole trajectory" (Sec. 1). We will revise the wording of the introduction to make this explicit.

Second, RL is the mainstream post-training method for long-horizon LLM agents. SFT relies on additional expert demonstrations and falls well behind RL. For example, with Qwen2.5-1.5B-Instruct on ALFWorld, SFT reaches 43.0% success, while RL training reaches 86.7% [2]. Therefore, we take RL-trained agents as the research scope of this paper.

Finally, we want to note that the instruct models perform poorly before task-specific post-training: all four evaluated models achieve at most 1.6% success on WebShop, and Qwen2.5-1.5B-Instruct achieves 0.0% on both ALFWorld ID and OOD. These near-zero success rates leave little room to measure further degradation under perturbations.

We also fine-tuned Qwen2.5-1.5B-Instruct using the 1,571 human-collected demonstrations released with WebShop as additional supervision. Its clean success rate (30.40%) is much lower than that of Vanilla GiGPO (57.00%). Although SFT has higher retention in some settings, its absolute success rate is generally lower than that of Vanilla RL.

WebShop, Qwen2.5-1.5BCleanGaussian (σ=.20)Dropout (p=.05)Structured (r=.20)Unstructured (r=.40)INT6
SFT30.40±3.3015.07±4.37 (49.6)14.40±5.36 (47.4)5.87±1.40 (19.3)21.13±4.15 (69.5)21.33±5.53 (70.2)
Vanilla GiGPO57.00±4.3918.73±3.03 (32.9)12.20±3.30 (21.4)2.27±1.86 (4.0)42.00±8.31 (73.7)45.20±9.18 (79.3)

Q2 / W1: How does SPrPO behave on larger models and on a non-Qwen model family?

We agree that evaluating SPrPO on larger models or other model families can strengthen the scope of our claims. To this end, we further trained Qwen2.5-3B, Qwen2.5-7B and Llama-3.2-3B-Instruct on WebShop. As shown in the tables below, SPrPO maintains its advantage on these models. We will include these results in the appendix of the revised paper.

Qwen2.5-3BCleanGaussian (σ=.20)Dropout (p=.05)Structured (r=.20)Unstructured (r=.40)INT6
Vanilla56.60±3.5226.40±4.61 (46.6)20.93±4.37 (37.0)15.80±7.30 (27.9)48.73±1.60 (86.1)55.73±4.45 (98.5)
Gaussian61.87±1.7232.80±2.96 (53.0)24.67±2.66 (39.9)17.20±3.70 (27.8)54.00±2.03 (87.3)61.07±2.14 (98.7)
SAM61.33±1.1727.87±3.93 (45.4)20.00±2.12 (32.6)15.07±3.11 (24.6)51.47±2.14 (83.9)60.67±1.63 (98.9)
SPrPO61.93±1.6836.60±5.37 (59.1)31.80±3.50 (51.3)20.87±5.22 (33.7)58.07±1.53 (93.8)61.00±1.31 (98.5)
Qwen2.5-7BCleanGaussian (σ=.20)Dropout (p=.05)Structured (r=.20)Unstructured (r=.40)INT6
Vanilla60.20±3.1742.80±3.70 (71.1)34.73±4.64 (57.7)32.20±5.23 (53.5)57.53±3.60 (95.6)59.00±2.31 (98.0)
Gaussian62.07±2.1645.67±3.26 (73.6)37.33±2.34 (60.1)27.27±7.00 (43.9)58.80±1.40 (94.7)61.20±1.31 (98.6)
SAM61.53±1.4044.73±3.06 (72.7)36.67±3.52 (59.6)29.87±7.96 (48.5)59.80±0.53 (97.2)60.40±1.00 (98.2)
SPrPO67.00±2.3150.47±3.07 (75.3)43.33±4.27 (64.7)42.33±6.72 (63.2)63.40±0.40 (94.6)66.27±0.58 (98.9)
Llama-3.2-3B-InstructCleanGaussian (σ=.20)Dropout (p=.05)Structured (r=.20)Unstructured (r=.40)INT6
Vanilla63.73±1.6027.73±2.00 (43.5)18.00±0.92 (28.2)22.27±0.81 (34.9)60.13±0.83 (94.4)55.27±1.67 (86.7)
Gaussian65.20±1.0634.47±2.10 (52.9)21.93±0.64 (33.6)27.53±0.61 (42.2)61.00±1.06 (93.6)60.47±1.63 (92.7)
SAM64.60±1.5931.13±2.00 (48.2)21.33±1.14 (33.0)16.27±1.45 (25.2)61.53±1.33 (95.2)61.20±0.53 (94.7)
SPrPO66.53±0.9041.87±1.79 (62.9)29.73±0.61 (44.7)35.47±2.01 (53.3)63.87±0.31 (96.0)63.40±1.71 (95.3)

Q3: Is SPrPO specific to GiGPO, and what changes are needed for other advantage estimators?

SPrPO is plug-and-play: its core logic does not depend on the advantage estimator. To verify this, we further trained SPrPO with GRPO on WebShop with Qwen2.5-1.5B. With GRPO, SPrPO remains the most robust method and also achieves the best clean success.

WebShop GRPOCleanGaussian (σ=.20)Dropout (p=.05)Structured (r=.20)Unstructured (r=.40)INT6
Vanilla-GRPO46.60±1.2212.20±1.97 (26.2)6.87±1.03 (14.7)4.60±1.39 (9.9)20.60±3.12 (44.2)30.00±1.59 (64.4)
Gaussian46.87±1.1713.47±2.21 (28.7)8.93±1.53 (19.1)1.93±0.61 (4.1)26.80±4.23 (57.2)33.07±2.04 (70.6)
SAM46.73±1.4013.93±1.50 (29.8)8.27±1.33 (17.7)2.13±0.42 (4.6)25.13±3.74 (53.8)31.07±1.36 (66.5)
SPrPO-GRPO48.80±0.5315.67±2.12 (32.1)10.00±1.56 (20.5)6.87±1.89 (14.1)30.27±2.91 (62.0)35.07±1.85 (71.9)

Q4(a): How is the bounded support required by Definition 1 and Theorem 2 reconciled with the Gaussian noise in Corollary 1 and the algorithm?

We acknowledge that this is an imprecision in our theoretical analysis, but it can be fixed simply with almost no change to the proofs. Specifically, let ν=N(0,Σ)\nu=\mathcal N(0,\Sigma) with Σ=diag(σ12,…,σd2)\Sigma=\mathrm{diag}(\sigma_1^2,\dots,\sigma_d^2) be the Gaussian in Corollary 1, and let the truncated Gaussian νδ\nu_\delta be ν\nu restricted to ∣ξi∣≤δσi|\xi_i|\le\delta\sigma_i for all channels ii. Then νδ\nu_\delta is bounded as required by Definition 1 and Theorem 2. Let pp be the probability that a sample from ν\nu violates this bound. The truncation inflates the probability of any event by at most a factor (1−p)−1(1-p)^{-1}, so Corollary 1 holds for νδ\nu_\delta with β\beta replaced by β/(1−p)\beta/(1-p). Since the Gaussian tail decays rapidly, pp is negligible for a moderate δ\delta, so the untruncated noise used in Algorithm 1 is practically indistinguishable from the truncated one.


Q4(b): Definition 1 measures deviation below the perturbed mean rather than below clean performance. Is this intended, and can the clean-to-mean gap be bounded?

Definition 1 characterizes downward deviations from the mean perturbed return, while SPrPO aims to raise that mean through policy optimization. With Jˉν(θ)=Eξ∼ν[J(πθξ)]\bar J_\nu(\theta)=\mathbb E_{\xi\sim\nu}[J(\pi_\theta^\xi)], the degradation from clean performance can be decomposed as

J(πθ)−J(πθξ)=J(πθ)−Jˉν(θ)⏟(I) clean-to-mean gap+Jˉν(θ)−J(πθξ)⏟(II) fluctuation around the perturbed mean.J(\pi_\theta)-J(\pi_\theta^\xi)=\underbrace{J(\pi_\theta)-\bar J_\nu(\theta)}_{\text{(I) clean-to-mean gap}}+\underbrace{\bar J_\nu(\theta)-J(\pi_\theta^\xi)}_{\text{(II) fluctuation around the perturbed mean}}.

We want to point out that SPrPO considers both terms. Optimizing the surrogate Lν\mathcal L_\nu (Definition 2) promotes improvement in the expected perturbed return Jˉν\bar J_\nu, aiming to reduce the clean-to-mean gap in term (I). However, improvement in expectation does not rule out degradation under some perturbations (Sec. 4.1). To address this, Theorem 1 provides a stability condition that controls variability in policy improvement, and Theorem 2 links stable updates to the concentration of perturbed returns characterized by Definition 1, corresponding to term (II). Empirically, Vanilla's collapse in Figure 4 appears as sharp cliffs under particular perturbations, illustrating the downward fluctuations captured by term (II).


Q4(c): How sensitive are the results to using m̂ as the proxy for the expected improvement?

Our ablation (Sec. 6.4, Table 9) shows that the m^\widehat m scheduling matters: with a fixed noise scale, clean success on WebShop drops from 60.00 to 52.13. This is because strong noise early in training makes successful trajectories rare, so the policy underfits. The m^\widehat m schedule avoids this by keeping the noise small early in training and raising it as learning progresses. The estimate of m^\widehat m can be more flexible: as shown in our response to Reviewer eC6B (W2(b)), estimating m^\widehat m by reusing the gradients of the update from θk\theta_k to θk+1\theta_{k+1} avoids the additional overhead and yields a similar scale and schedule, while SPrPO retains its advantage.

Q5: Why perturb only the FFN intermediate channels?

We acknowledge that extending the perturbation to other modules is a meaningful ablation. SPrPO itself is not specific to FFN; in the main experiments, we perturb only the FFN channels for simplicity, consistent with prior work on structured pruning of FFN intermediate channels [3], with FFN layers comprising about 88% of the per-layer linear parameters in Qwen2.5-1.5B.

To study the effect of the perturbation target, we further applied SPrPO to both the FFN and the attention Q/K/V projections on WebShop with Qwen2.5-1.5B, as shown in the table below. Perturbing attention as well yields better retention than Vanilla, but worse overall performance than perturbing FFN only, possibly because perturbing attention introduces more instability into training.

WebShopCleanGaussian (σ=.20)Dropout (p=.05)Structured (r=.20)Unstructured (r=.40)INT6
Vanilla57.00±4.3918.73±3.03 (32.9)12.20±3.30 (21.4)2.27±1.86 (4.0)42.00±8.31 (73.7)45.20±9.18 (79.3)
SPrPO (FFN)60.00±0.7230.13±1.81 (50.2)20.87±2.00 (34.8)12.40±3.65 (20.7)50.20±2.99 (83.7)55.67±1.60 (92.8)
SPrPO (FFN+QKV)51.00±1.8022.27±4.36 (43.7)15.87±3.56 (31.1)8.80±4.98 (17.3)39.40±7.41 (77.3)43.60±7.98 (85.5)

Q6(a): Given the large seed variance on WebShop, how reliable are the key claims?

We want to point out that this level of variance is common for RL-trained LLM agents in our tasks, due to the stochasticity of long-horizon agent tasks. The GiGPO paper [4] also reports comparable standard deviations, e.g., 4.7 on ALFWorld and 4.5 on WebShop with Qwen2.5-1.5B. Besides, SPrPO has a smaller standard deviation than Vanilla in most settings, which further validates the efficacy of our method.


Q6(b): How does SPrPO compare with NPFT and QeRL, and what is the novelty boundary?

We acknowledge that QeRL and NPFT have inspired our work, but SPrPO differs substantially from them in the task setting, the training goal, the perturbations considered, and the design of the injected noise, as shown in the table below.

QeRLNPFTSPrPO (ours)
Training paradigmRLSupervised fine-tuningRL
Task settingSingle-turn reasoningStatic language benchmarksLong-horizon, multi-turn agents
Training goalEfficient RL and quantization-aware training (QAT)Post-training robustness (PTR) to quantizationPTR to unknown deployment perturbations
Perturbations studiedQuantizationQuantizationGaussian noise, dropout, structured and unstructured pruning, quantization
Injected noiseEqual variance across channelsRandom noise on sensitive weightsTraining-stability-aware: less noise on sensitive channels

Because the task setting and training objectives differ substantially, we adapt their noise-injection principles to the same agent RL pipeline and FFN-channel perturbation space. Our SAM-style baseline concentrates perturbations along sensitive directions, adapting NPFT's principle of perturbing sensitive components. Our Multiplicative Gaussian baseline applies sensitivity-independent channel noise, corresponding to the equal-variance noise design in QeRL. These adaptations provide a controlled comparison of noise-design choices within the same task and RL training setup.

Notably, NPFT and SPrPO allocate the noise in opposite ways. NPFT injects noise on the sensitive weights, whereas SPrPO keeps the noise mild on sensitive channels. This difference stems from the different tasks. NPFT targets the basic language ability of the LLM, measured by perplexity on static text, while we target the ability to complete long-horizon agentic tasks. A perturbation with little effect on the language ability can still cause the agent to fail. Concentrating the noise on sensitive channels therefore destabilizes RL training of agents, as shown in Figure 5(a), which is the core intuition of our noise design.

To clarify the novelty boundary, we summarize how SPrPO differs from these works in three aspects:

  • To our knowledge, we are the first to study perturbation robustness in the setting of long-horizon, multi-turn LLM agents.
  • We target post-training robustness to unknown perturbations, rather than a specific type of perturbation such as quantization.
  • We design our noise allocation mechanism based on our theoretical analysis.
Reviewer_eC6BOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines how RL-trained language-model agents degrade under changes to their policy, including hidden-channel noise, dropout, pruning, and quantization. A TRPO-based analysis relates reliable policy improvement to the variability induced by perturbations. SPrPO uses that motivation to inject multiplicative Gaussian noise into FFN channels, reducing noise on channels estimated to be sensitive and scaling it with a gradient-norm proxy for improvement. Experiments use Qwen2.5-1.5B-Instruct with GiGPO on ALFWorld and WebShop and report both clean success and performance under deployment-time perturbations.

Critical strengths
  1. The empirical question is practical and the evaluation goes beyond the exact noise used during training. SPrPO improves robustness to weight quantization and SparseGPT pruning as well as FFN-channel perturbations. On WebShop, INT6 success is 55.67% versus 45.20% for vanilla training, and Gaussian-noise success at sigma = 0.20 is 30.13% versus 18.73%, while clean success also improves (Table 1). Reporting retention relative to each method's clean policy helps separate robustness from simply starting with a better policy.

  2. The component ablations are informative. Removing sensitivity adaptation or the gradient-norm schedule hurts clean success, and the full method gives the best WebShop mean in each column of Table 9. Three-seed reporting, full perturbation curves, and the explicit timing breakdown in Table 4 make the practical contribution reasonably assessable.

Critical weaknesses
  1. The theory motivates the algorithm but does not provide the stated stability guarantee for its actual updates. Corollary 1 requires expected surrogate sensitivities, a bound on maximum policy drift, and a positive improvement lower bound. Section 5 replaces these with gradients from one reused batch and a squared gradient norm, without checking the sufficient condition. The convergence theorem also assumes a sufficiently concentrated initial perturbed return; it should not be read as showing that an arbitrary fragile initial policy becomes robust. Definition 1 measures concentration around the perturbed mean, which can itself be poor, rather than preservation relative to clean performance. The empirical retention measurements are therefore more relevant to deployment than the theoretical robustness definition.

  2. The ablation does not fully isolate sensitivity allocation from the total injected noise. Setting alpha = 0 changes both how noise is distributed and its aggregate scale. A matched-noise-energy isotropic or shuffled-sensitivity control would more directly test the proposed mechanism. This distinction matters because the evidence is restricted to one 1.5B model and one underlying RL algorithm, and the method costs about 1.67x and 1.78x the vanilla training time per step on ALFWorld and WebShop, respectively (Table 4). The current results establish gains at equal training steps, but not whether the gains persist under an equal wall-clock budget.

Questions for authors
  1. Does SPrPO still outperform an isotropic baseline and a shuffled-channel sensitivity baseline when their total noise energy and training budget are matched? This is the control most likely to change my view of the allocation mechanism.
  2. How are near-zero sensitivity estimates handled in the inverse-sensitivity rule? Please specify any floors, clipping, normalization, or smoothing, and show the actual noise scales and policy divergence over training. Those details are needed to reproduce the intended stability behavior.
  3. Can you distinguish the guarantees of the ideal update sequence from the heuristic guarantees, if any, of Algorithm 1? In particular, is the sufficient condition ever checked or approximately satisfied in the reported runs?
4 comments

We thank the reviewer for the careful reading and the concrete suggestions. We address each question and weakness below and will incorporate the clarifications and new results into the revised version.

Q1: Does SPrPO still outperform isotropic noise when the total noise energy is matched?

Yes, and we will clarify an implementation detail and a controlled ablation as follows.

First, the total noise energy is independent of the allocation. In our actual implementation, we set σi2∝S^i−α\sigma_i^2\propto\widehat S_i^{-\alpha} and then normalize the weights to unit mean within each layer (more implementation details are clarified in Q2). The per-layer noise energy ∑iσi2\sum_i\sigma_i^2 is therefore governed by ρ and m^\widehat m, and α only changes how this energy is distributed across channels.

Second, the allocation is the key design. Figure 5(a) shows that, at the same perturbation magnitude, perturbations concentrated on sensitive channels (SAM-style) drive task success to zero, whereas sensitivity-aware noise substantially limits the degradation. Protecting sensitive channels to preserve training stability is the core intuition of our noise design.

Finally, an ablation with strictly controlled noise energy. We compared three allocations on WebShop: SPrPO (α=1), isotropic (α=0), and reversed (α=−1, more noise on sensitive channels). We disabled the m^\widehat m scheduling (keeping σ=0.2 fixed throughout training), so that the comparisons are strictly controlled to have the same total noise energy. The results below show great performance degradation without proper noise allocation.

MethodCleanGaussian (σ=.20)Dropout (p=.05)Structured (r=.20)Unstructured (r=.40)INT6
α=1 (SPrPO)52.13±0.4226.40±4.7019.33±5.323.60±2.8845.00±4.2049.73±2.42
α=0 (isotropic)36.07±0.4217.60±4.6913.80±2.993.53±2.2029.60±5.5734.33±6.77
α=−1 (reversed)7.07±0.425.07±1.144.07±0.232.40±0.727.33±1.037.07±0.50

Q2: How are near-zero sensitivities handled? Clarification of the implementation details, the actual noise scale and the policy drift statistics.

We first clarify the implementation details, then report the noise scale and policy drift observed during training.

First, implementation details. Within each layer ℓ\ell, the inverse-sensitivity rule is implemented as follows:

MechanismImplementation
Safety thresholdA safety threshold, (S^i/Sˉℓ+0.01)−α(\widehat S_i/\bar S_\ell+0.01)^{-\alpha} with the layer mean Sˉℓ\bar S_\ell, avoids numerical issues for small sensitivities; dividing by Sˉℓ\bar S_\ell makes it independent of the gradient scale.
Noise-variance normalizationThe resulting weights are normalized to unit mean within each layer, so that the mean noise variance in each layer equals ρ2m^2\rho^2\widehat m^2. This keeps the total noise energy invariant to the allocation.
ClippingEach channel's noise variance is clipped to between 0.25 and 2 times the layer average to avoid vanishing or excessively large noise.

Second, the actual noise scale and policy drift. During SPrPO training, the global noise scale σeff\sigma_{\mathrm{eff}} is 0.081/0.40 (mean/max) on ALFWorld and 0.141/0.40 on WebShop, and the KL divergence between the noisy rollout policy and the clean policy is 8.7×10⁻⁴/6.1×10⁻³ on ALFWorld and 7.6×10⁻⁴/7×10⁻³ on WebShop. We will add these details and the per-step curves to the appendix.

Q3: Which guarantees hold for the ideal update sequence and which for Algorithm 1, and is condition (8) satisfied in practice?

Thank you for highlighting this missing step in our explanation. Algorithm 1 is designed to satisfy condition (8) under the gradient-based estimates S^i\widehat S_i and m^\widehat m, which we will clarify theoretically and validate empirically.

(a) Guarantees for the ideal update sequence. For the ideal update sequence satisfying condition (8), Corollary 1 guarantees stable policy improvement with probability at least 1−β1-\beta, i.e., Pr⁡ξ∼ν(ΔJξ≥0)≥1−β\Pr_{\xi\sim\nu}(\Delta J_\xi\ge0)\ge1-\beta, which extends the monotonic improvement of TRPO to the perturbation setting.

(b) Estimated condition and the design of Algorithm 1. In practice, the exact quantities in (8) are difficult to evaluate in RL "because the data distribution changes with the policy" (Sec. 5). We therefore estimate them with gradients on the current rollout batch, giving S^i\widehat S_i and m^\widehat m (Sec. 5). Under these estimates, condition (8) reads

∑iσi2S^i≤β1−βm^2.\sum_i \sigma_i^2 \widehat S_i \le \frac{\beta}{1-\beta} \widehat m^2 .

To satisfy this condition, Algorithm 1 sets the noise variance of each channel as σi2∝m^2/S^i\sigma_i^2\propto\widehat m^2/\widehat S_i. Below we show theoretically in (c) and empirically in (d) that Algorithm 1 satisfies this condition.

(c) Theoretical analysis.

Let dℓd_\ell be the number of channels in layer ℓ\ell. With α=1, channel ii in layer ℓ\ell receives the noise variance

σi2=ρ2m^2⋅S^i−1dℓ−1∑j∈ℓS^j−1.\sigma_i^2 = \rho^2 \widehat m^2 \cdot \frac{\widehat S_i^{-1}}{d_\ell^{-1}\sum_{j\in\ell}\widehat S_j^{-1}} .

Next, summing first over the channels of each layer and then over all layers gives

= \rho^2\widehat m^2\sum_\ell \frac{d_\ell}{d_\ell^{-1}\sum_{j\in\ell}\widehat S_j^{-1}} = \rho^2\widehat m^2 \kappa \widehat S,$$ where $\widehat S=\sum_i\widehat S_i$ and $$\kappa = \frac{\sum_\ell d_\ell\left(d_\ell^{-1}\sum_{j\in\ell}\widehat S_j^{-1}\right)^{-1}}{\sum_\ell d_\ell\left(d_\ell^{-1}\sum_{j\in\ell}\widehat S_j\right)} \in(0,1] .$$ The denominator of κ equals $\widehat S$. κ is the ratio between the harmonic and arithmetic means of the sensitivities within each layer, and it measures how much the sensitivity-aware allocation reduces the left-hand side of (8): it equals 1 when all channels in a layer are equally sensitive and decreases as the sensitivities become more uneven. Therefore, substituting this into the estimated condition (8), condition (8) is equivalent to $$\rho^2 \kappa \widehat S \le \frac{\beta}{1-\beta} .$$ **(d) Empirical verification.** We verified this condition on the WebShop run with Qwen2.5-1.5B. The per-channel sensitivity $\widehat S_i$ is small, about $2.3\times10^{-5}$ on average, so $\widehat S$ summed over all 250,880 FFN channels is only about 5.74. With the estimated $\kappa\approx0.39$, the left-hand side of the condition is $\rho^2\kappa\widehat S \approx 0.2^2\times0.39\times5.74 \approx 0.09$, which is below the threshold $\beta/(1-\beta)$ for both β=0.2 (0.25) and β=0.1 (0.111). Tracked over the whole training run, condition (8) is satisfied at 98% of the steps for β=0.2 and 85% for β=0.1. Note that $\beta$ is the tolerated probability of degradation, and based on our estimates, most training steps should improve performance with high probability, which can explain the good performance of SPrPO.

W1: Clarifications on the theoretical results

W1(a): Stability guarantee for the actual updates of Algorithm 1. Please see our response to Q3.

W1(b): Assumption 1 of Theorem 2. We agree that Theorem 2 should not be read as turning an arbitrary fragile policy into a robust one. SPrPO focuses on the optimization process rather than its starting point, and Theorem 2 characterizes this process: it shows that a sequence of stable updates converges to perturbation robust policies (Sec. 4.2). Assumption 1 only fixes the reference point from which the analysis of this sequence starts, and this point can be any iterate πK\pi_K, since the subsequent updates have the same limit points; we state it for π0\pi_0 for ease of presentation. By the way, the initial policy of RL training often performs similarly poorly across nearby perturbations, so its perturbed return tends to have a small variance.

W1(c): Definition 1 and preservation relative to clean performance. Definition 1 characterizes downward deviations from the mean perturbed return, while SPrPO aims to raise that mean through policy optimization. With Jˉν(θ)=Eξ∼ν[J(πθξ)]\bar J_\nu(\theta)=\mathbb E_{\xi\sim\nu}[J(\pi_\theta^\xi)], the degradation from clean performance can be decomposed as

J(πθ)−J(πθξ)=J(πθ)−Jˉν(θ)⏟(I) clean-to-mean gap+Jˉν(θ)−J(πθξ)⏟(II) fluctuation around the perturbed mean.J(\pi_\theta)-J(\pi_\theta^\xi)=\underbrace{J(\pi_\theta)-\bar{J}_\nu(\theta)}_{\text{(I) clean-to-mean gap}}+\underbrace{\bar{J}_\nu(\theta)-J(\pi_\theta^\xi)}_{\text{(II) fluctuation around the perturbed mean}}.

We want to point out that SPrPO considers both terms. Optimizing the surrogate Lν\mathcal L_\nu (Definition 2) promotes improvement in the expected perturbed return Jˉν\bar J_\nu, aiming to reduce the clean-to-mean gap in term (I). However, improvement in expectation does not rule out degradation under some perturbations (Sec. 4.1). To address this, Theorem 1 provides a stability condition that controls variability in policy improvement, and Theorem 2 links stable updates to the concentration of perturbed returns characterized by Definition 1, corresponding to term (II). Empirically, Vanilla's collapse in Figure 4 appears as sharp cliffs under particular perturbations, illustrating the downward fluctuations captured by term (II).


W2: Separating allocation from total noise energy, and training cost

W2(a): Allocation at matched noise energy. Please see our response to Q1.

W2(b): Training cost. We agree that the training overhead is a limitation of SPrPO as presented in the paper. SPrPO estimates the channel sensitivities S^i\widehat S_i and the expected improvement m^\widehat m at θk+1\theta_{k+1} from gradients, which requires an additional forward/backward pass after each policy update and increases the per-step training time.

To reduce this cost, we implement a lite version of SPrPO that reuses the gradients computed while the policy is updated from θk\theta_k to θk+1\theta_{k+1}, which removes the additional forward/backward pass. Intuitively, each update changes the parameters only slightly, so statistics such as the channel sensitivities and the expected improvement vary smoothly across consecutive updates. To validate this design, we retrained SPrPO with gradient reuse for 200 steps on WebShop with Qwen2.5-1.5B:

MethodCleanGaussian (σ=.20)Dropout (p=.05)Structured (r=.20)Unstructured (r=.40)INT6
Vanilla57.00±4.3918.73±3.03 (32.9)12.20±3.30 (21.4)2.27±1.86 (4.0)42.00±8.31 (73.7)45.20±9.18 (79.3)
SPrPO60.00±0.7230.13±1.81 (50.2)20.87±2.00 (34.8)12.40±3.65 (20.7)50.20±2.99 (83.7)55.67±1.60 (92.8)
SPrPO (w/ gradient reuse)58.53±0.9229.20±1.51 (49.9)21.80±2.72 (37.2)19.20±2.12 (32.8)48.07±3.82 (82.1)53.47±2.19 (91.4)

Gradient reuse retains SPrPO's advantage while saving about 30% of the training time compared with the full implementation. The remaining overhead mainly comes from noise injection, which is an element-wise operation and can be further optimized. The m^\widehat m estimated with gradient reuse has a similar mean (1.12 vs. 0.97) and plays the same scheduling role as described in Sec. 6.4: it stays close to zero early in training and grows later.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.