acceptodds
Under review as a conference paper at ICLR 2027

Learning to Act under Noise: Enhancing Agent Robustness via Noisy Environments

Abstract

Recent advances in large language models (LLMs) have facilitated the widespread deployment of LLMs as interactive agents capable of reasoning, planning, and tool use. Despite strong performance on existing benchmarks, such agents often exhibit notable degradation when deployed in real-world settings, where environments are inherently stochastic and imperfect. We argue that this discrepancy arises from a fundamental mismatch between idealized training settings and real-world interaction dynamics, where current paradigms rely on carefully curated task instructions and stable, well-controlled environments. To address this gap, we propose \ours, an agentic training framework that explicitly incorporates environmental imperfections into the agent learning process. We identify two major sources of interaction noise in real-world scenarios: user noise, which captures ambiguity and variability in user interaction, and tool noise, which reflects failures and anomalies in tool execution. We introduce such perturbations into the training pipeline by modifying user interaction patterns and simulating tool execution results within the training environment. To stabilize training while encouraging agents to handle increasingly challenging imperfections, noise is applied to only a subset of rollouts and progressively increased in difficulty as the model adapts to the current noise level. Extensive experiments demonstrate that our approach consistently improves agent robustness under noisy and dynamic environments. Our analysis reveals that training under noise conditions also yields performance gains on idealized benchmarks, suggesting that controlled exposure to environmental noise promotes more generalizable reasoning and decision-making behaviors. Our findings highlight the importance of modeling interaction imperfections for bridging the gap between agent training and real-world deployment.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Agent, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_oMkJOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper proposes NoisyAgent, an agentic reinforcement learning framework designed to improve LLM agent robustness under noisy environments. It introduces user-side and tool-side perturbations during training and employs a hybrid training strategy that combines clean and noisy rollouts with group-wise advantage normalization. An adaptive scheduling strategy progressively increases noise difficulty and proportion based on the performance gap between clean and noisy environments. Experiments on AgentNoiseBench and standard agent benchmarks demonstrate improvements in both noisy and idealized environments.

Critical strengths
  1. Practical motivation: The paper addresses an important limitation of existing agentic RL training, which typically assumes idealized environments. Incorporating realistic interaction imperfections into training is a meaningful direction for improving agent robustness.
  2. Effective noise-aware training: The proposed approach consistently improves performance across different benchmarks and model scales, demonstrating the benefits of incorporating noise into agentic RL training.
Critical weaknesses
  1. Definition of noise severity: The definition of the “severity” of each noise type is unclear. Noise severity could be defined by how much a perturbation changes the original user instruction or tool outcome, or by how difficult the resulting task becomes. However, the paper lacks clear mechanisms for controlling noise severity during generation and validating the severity of generated perturbations. This makes it unclear how noise difficulty is progressively increased during scheduling.
  2. Realism of noise: The authors do not empirically evaluate how realistic the generated noise is. Without such validation, although the proposed method demonstrates strong performance in the specific experimental settings, it remains unclear whether the generated perturbations adequately reflect real-world interaction patterns and whether the resulting robustness improvements would generalize to real-world environments.
  3. Details of scheduling: Although the authors define noise scale and describe the criteria for increasing noise difficulty, the exact update rules are not sufficiently specified. In particular, the initial noise configurations, increment sizes, and update frequency remain unclear. Providing these details would improve the clarity and reproducibility of the proposed method.
  4. Effectiveness of scheduling: The results reported in Table 3 show that joint training without scheduling achieves relatively comparable performance to the full method. Although additional experiments in Appendix B.2 show consistent improvements, the incremental benefits of adaptive scheduling appear relatively modest compared with those of noise injection itself. Additionally, the fixed noise difficulty and noise proportion used in the ablation are not reported, making it difficult to assess the effectiveness of the proposed scheduling strategy.
  5. Ablation on noise types: Overall, training with noise improves final performance. However, the individual contributions of different noise types to the performance improvements remain unclear. In particular, the paper does not provide separate ablation studies for user-side and tool-side noise, making it difficult to assess the effectiveness of each component.
  6. Validation of task solvability: Although the authors claim that noise injection preserves task solvability, they do not explain how this property is ensured or verified. Providing details on the verification procedure would improve the clarity and reproducibility of the proposed method.
Questions for authors
  1. How is the severity of each noise type defined and quantified? How do the authors control and validate noise severity during generation?
  2. How do the authors ensure that the generated noise realistically reflects real-world user interactions and tool failures? Could the authors provide empirical validation, such as human evaluations or comparisons with real-world interaction data?
  3. Could the authors clarify the exact scheduling procedure, including how noise scale and difficulty are updated when the progression criterion is satisfied? What are the initial values, increment sizes, and update frequency?
  4. Given the relatively modest performance improvements from scheduling in Table 3 and Appendix B.2, could the authors provide further evidence of the effectiveness of adaptive scheduling? Also, what fixed noise difficulty and proportion were used in the w/o scheduling ablation?
Reviewer_vtg9OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Noisy agent, which is an agentic rl method that inject user side noise such as ambiguous, incosistent, or redundant instructions and tool related failures into a subset of training rollouts.

Critical strengths
  1. the idea is simple and practical
  2. the gains are consistent. Noisyagent beats baseliens on all domains, model sizes, and settings.
  3. the paper is well written
Critical weaknesses
  1. my biggest concern is that the statistical evidence is weak. currenlty, each method seems to come from a single training run, with no seeds, variance, or confidence intervals. Serveral gains currenlty are around 1 point, which may be within run-to-run noise.
  2. another thing is that it looks like that training noise closely matches the test noise. So the injected categories mirror the agentnosiebench taxonomy, which means most of the noisy-benchmark gain may be in distribution.
  3. I think the paper is m issing key ablations. Currently, "w/o controlled injection" puts noise on 100% of rollouts and scores below the untraied base model on noisy delivery. The natural baselines however are untested: a fixed 50% nosise ratio with standard joint normalization, and group-wise versus joint advantage normalization alone.
  4. the clean benchmark gain is not explained. a control with equally hard clean tasks would show whether noise specifically matters.
  5. I think the perturbation validity is not checked. There is no huamn check that perturbed tasks stay solvable.
Questions for authors

See above; Happy to discuss in rebuttal :)

Reviewer_g6G8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes NoisyAgent, a reinforcement learning framework for improving the robustness of LLM agents through training in perturbed environments. The framework introduces user-side perturbations, such as ambiguity and inconsistent interactions, and tool-side perturbations, such as execution errors and corrupted responses. Training combines clean and noisy rollouts, computes advantages separately within these groups, and uses a curriculum that increases noise exposure and difficulty according to the performance gap between clean and noisy conditions.

Experiments with Qwen3-8B and Qwen3-32B compare the approach against GRPO, DAPO, and GSPO on AgentNoiseBench and the corresponding standard benchmarks. The reported results show improvements under noisy conditions and smaller, consistent gains under clean conditions. Additional analyses examine component ablations, noise-ratio caps, held-out noise categories, computational overhead, and interaction behavior.

Critical strengths

S1.A concrete training approach addressing a relevant limitation of interactive agents: The paper moves from documenting robustness failures to training agents under imperfect interactions. Combining clean and noisy rollouts, adapting noise exposure, and accounting for the different reward distributions of the two rollout groups form a coherent approach to this problem. The framework is compatible with existing agentic RL infrastructure and addresses practical sources of failure in user interactions and tool responses.

B.Consistent improvements supported by experiments across model sizes and domains: In Tables 1 and 2, NoisyAgent achieves higher Avg@4 than GSPO across all six domains and both backbone sizes under both noisy and clean conditions. For example, on noisy Delivery tasks, Avg@4 increases from 16.00 to 21.50 for Qwen3-8B and from 23.75 to 28.75 for Qwen3-32B. The extended ablations in Table 11 show that the benefit of mixing clean and noisy rollouts is observed beyond the initial Qwen3-8B Delivery setting. These results provide grounded evidence for the effectiveness of the complete training pipeline.

Critical weaknesses

W1.The adaptive training procedure is insufficiently specified, and its distinct contribution is not fully isolated: Section 3.2 and Appendices A.1 and B.3 provide the noise-ratio cap, scheduling threshold, and a safeguard against advancing when the noisy group has no successful rollouts. However, they do not specify several central implementation choices: the warm-up duration and transition to a nonzero noise ratio, the available severity levels and their increments, and how the performance gap is aggregated and used to update each noise type. Without these details, it is difficult to reproduce the curriculum or determine the actual noise exposure experienced during training. The ablations provide useful evidence, but their interpretation would benefit from clearer controls. The “w/o scheduling” variant uses perturbations of fixed complexity, yet its noise-ratio configuration and total exposure relative to the adaptive method are not sufficiently described. Separate advantage normalization in Equation 10 is also not evaluated independently, despite its stated role in stabilizing optimization. Across Tables 3 and 11, the scheduling-related gains in noisy Avg@4 are approximately 1–2 percentage points. Clearer baseline specifications and uncertainty estimates are therefore particularly relevant to establishing the value of the adaptive mechanism. Complete pseudocode and configuration details, together with targeted evidence about normalization and scheduling, would strengthen the technical contribution.

W2.The operational definitions and validation of the perturbations leave the scope of the robustness claims uncertain: Section 3.1 describes training-time “Misleading” tool noise as responses containing incorrect or inconsistent information. Appendix B.1 nevertheless treats “Erroneous Outputs” as a category excluded from training. The referenced benchmark distinguishes erroneous factual outputs from misleading signals, but the manuscript’s description does not make that distinction operationally clear. The two categories may differ in the implementation, but the generator prompts and representative examples are needed to establish this. I appreciate that Appendix B.1 explicitly separates training and evaluation instances and uses different noise generators. This supports transfer across instances and generators. The stronger claim of transfer to an unseen noise category additionally requires a clear account of what was excluded during training.

W3. Another related issue concerns task solvability. Section 3.1 states that preserving the underlying objective prevents perturbations from invalidating the task, while Appendix C.3 acknowledges that generated disturbances can still invalidate solution paths. Preserving the objective alone does not establish that an interaction remains recoverable. For example, when a tool response is replaced with an error, it matters whether the underlying state-changing operation has already occurred. The paper should describe the relevant validity checks and clarify the relationship between response corruption and actual tool execution. This would help distinguish robustness to valid interaction disturbances from adaptation to particular simulator artifacts.

Questions for authors

Q1.Perturbation definitions and validity: How does training-time “Misleading” noise differ operationally from the held-out “Erroneous Outputs” category? Please provide the relevant generator instructions and representative examples. What checks establish that the generated perturbations preserve a valid solution path, particularly for inconsistent user interactions and failures involving state-changing tools?

Q2.Adaptive training and component attribution: Could you provide a complete specification of the curriculum, including initialization, severity levels, update increments, and aggregation of the clean–noisy performance gap? What noise ratio and severity are used in the “w/o scheduling” baseline, and what evidence isolates the benefit of separate advantage normalization from the other components?

Q3.Reliability of the reported gains: Can you report uncertainty for the main comparisons and scheduling ablations using the available task-level outcomes, and define the shaded bands in Figure 2? Please distinguish evaluation variability from variability across independently trained models. This would help assess the smaller improvements attributed to scheduling and the gains in clean environments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.