- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how policy perturbations affect RL-trained LLM agents and aims to make these agents more robust when solving tasks. The authors propose Stable Perturbation-Robust Policy Optimization (SPrPO). SPrPO injects adaptive, sensitivity-aware Gaussian perturbations into the FFN channels during training. Experiments on ALFWorld and WebShop cover several types of policy perturbations, including Gaussian noise, dropout, structured and unstructured pruning, and quantization, and SPrPO generally outperforms the baselines.
Critical strengths
- The problem is of practical significance. Deploying large models is a challenge in many real-world scenarios, and compression or changes in numerical precision at deployment time can degrade the performance of RL-trained agents.
- Broad evaluation across perturbations. The experiments cover five types of perturbations, each at multiple strengths, and the results show that SPrPO generally outperforms the baselines.
Critical weaknesses
-
Missing comparison with directly training a smaller model. The paper is motivated by the observation that models are often compressed at deployment time, which degrades task performance. The authors therefore propose this framework to improve robustness. However, I have a question: how would a smaller model, such as Qwen2.5-0.5B-Instruct, perform if it were directly trained and deployed? If such a small model already performs well, would the proposed method still be necessary?
-
Missing relevant baselines. The paper discusses two closely related works, NPFT and QeRL, but does not compare with either of them.
Questions for authors
1.How does a directly trained smaller model (e.g., Qwen2.5-0.5B-Instruct) perform compared with the compressed Qwen2.5-1.5B-Instruct policies trained with SPrPO?
2.How does SPrPO compare with NPFT and QeRL? If such a comparison is not feasible, could the authors explain why?
3.Under settings such as dropout and structured pruning, SPrPO’s success rates remain very low compared with the clean setting. How do the authors view the practical value of the method in these settings?
4 comments
Q3: What is the practical value of SPrPO?
Although SPrPO still faces large performance drops under the most severe perturbations, it can offer practical benefits for deploying RL-trained agents, as discussed below.
First, our work studies general policy perturbations, which in real-world deployment are not limited to model compression and also arise from different hardware, numerical precision and random noise. For example, RL frameworks such as verl [2] train the LoRA adapter in FP32 and often merge it into the BF16 base model for serving, where the rounding of the learned update perturbs the trained policy. This is the typical pipeline when a small model is fine-tuned with RL for a specific application, such as online shopping, household assistance or mobile device control, which makes robustness to post-training perturbations particularly valuable in routine deployment.
Second, even when a perturbation severely degrades a policy, a more robust policy is a better starting point for recovery. In Q1 / W1, pruning 80% of the FFN channels causes a large performance drop for both 1.5B policies, but after the same lightweight recovery, the pruned SPrPO policy recovers to a much higher level than the pruned Vanilla policy.
References
[1] Xinyin Ma, Gongfan Fang, and Xinchao Wang. LLM-Pruner: On the Structural Pruning of Large Language Models. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
[2] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. HybridFlow: A Flexible and Efficient RLHF Framework. In Proceedings of the European Conference on Computer Systems (EuroSys), 2025.
I thank the authors for the response and additional experiments, which address most of my concerns.
Thank you for your reading, and we are pleased to see that most of your concerns have been addressed. We are also very grateful for your insightful and inspiring questions: they led us to the prune-and-recover experiment, whose results were a pleasant surprise to us, and we plan to study this direction further in follow-up work.
If any point remains open, please let us know, and we would greatly appreciate it if you could reconsider your assessment based on the additional experiments. Thank you again for your time and constructive feedback.
We thank the reviewer for the practical questions. We address each question and weakness below and will incorporate the clarifications and new results into the revised version.
Q1 / W1: How does a directly trained Qwen2.5-0.5B compare with a compressed Qwen2.5-1.5B policy trained with SPrPO?
When a compact policy is needed for deployment, directly training a smaller model is a natural alternative to compressing a larger RL-trained policy, and we compare the two below.
First, quantization causes relatively less degradation to the policy, and it is a widely used model compression approach in practice. The SPrPO policy with INT6 quantization outperforms the directly trained Qwen2.5-0.5B by a large margin.
Second, directly pruning the 1.5B policy to the size of Qwen2.5-0.5B is challenging: it requires removing more than 80% of the FFN channels, which leads to a severe performance collapse. In practice, however, pruning is commonly followed by lightweight recovery training on a small amount of data [1]. Following this pipeline, we pruned 80% of the FFN channels of both 1.5B policies with the criterion of LLM-Pruner, which gives 0.62B parameters, and recovered them by distillation from their own unpruned policies on only 500 trajectories.
Surprisingly, with this lightweight recovery, the pruned SPrPO policy outperforms the directly trained Qwen2.5-0.5B by a large margin at a comparable size, as well as the pruned Vanilla policy. We will include this comparison as supplementary results in the appendix.
Finally, we want to clarify the scope of this paper. Model compression is an important observation and motivation of our work. Our research focus is whether an RL-trained agent remains reliable when facing unknown perturbations, and model compression is one typical such perturbation in deployment. The practical value of SPrPO therefore extends beyond compression, which we explain in more detail in Q3.
Q2 / W2: How does SPrPO compare with NPFT and QeRL?
Thank you for pointing to these works. We acknowledge that QeRL and NPFT have inspired our work, but SPrPO differs substantially from them in the task setting, the training goal, the perturbations considered, and the design of the injected noise, as shown in the table below.
Because the task setting and training objectives differ substantially, we compare their noise-design principles within the same agent RL pipeline (Sec. 6.4). Our Multiplicative Gaussian baseline uses sensitivity-independent noise, corresponding to QeRL's equal-variance design, while our SAM-style baseline concentrates perturbations on sensitive channels, adapting NPFT's principle of perturbing sensitive components. We will clarify these connections in the revised version.