acceptodds
Under review as a conference paper at ICLR 2027

Toward Optimal Adversarial Attacks in Reinforcement Learning via Rate–Distortion Theory

Abstract

Reinforcement learning (RL) has been deployed in many security-related applications. To improve the robustness of RL systems, it is important to systematically study potential adversarial attacks. Most prior work has considered deterministic adversarial attack strategies targeting either the agent or the environment, which can often be mitigated or reversed by defensive RL agents. In this paper, we propose a provably "uncounterable" class of adversarial attacks on RL systems. Our attack leverages a rate-distortion information-theoretic approach to stochastically perturb the agents' observations of the underlying Markov Decision Process (MDP). These perturbations are designed to strongly or even entirely limit the agent's ability to obtain accurate information about the true environment. This proposed framework is modular and can be integrated with various observation attack strategies targeting different components of the MDP, including state, actions, transition functions, or reward signals. We derive an information-theoretic lower bound on the victim agent's reward regret and show the impact of rate-distortion attacks on state-of-the-art algorithms.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning, information theory, adversarial attacks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SH5jOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes an information-theoretic rate-distortion framework for adversarial attacks against reinforcement learning agents, aiming to reduce the information that perturbed observations reveal about the true environment under a distortion budget.

Critical strengths
  1. The authors provide a relatively comprehensive theoretical analysis.

  2. The authors formulate and quantify perturbations from an information-theoretic perspective.

Critical weaknesses
  1. The writing needs further improvement. For example, the motivating example in the Introduction is not easy to follow.

  2. The paper provides only theoretical results and lacks experimental validation.

  3. The types of perturbations considered lack a clear connection to the physical world.

Questions for authors
  1. Does the type of environmental uncertainty defined in this paper exist in practical settings? In standard MDPs and methods such as SA-MDP, state transitions are also stochastic rather than deterministic. How does the notion of uncertainty in this paper differ from that considered in these existing works?

  2. In methods such as SA-MDP and PA-AD, perturbations are typically constrained by an L∞L_\infty norm with a small magnitude, allowing them to be regarded as noise. Does the budget-constrained attack proposed in this paper ensure that the perturbations remain reasonable and imperceptible?

  3. Can the effectiveness of the proposed method be validated through practical experiments, for example, in simple RL environments such as Atari?

Reviewer_bV56OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes an information-theoretic framework for attacking RL, in which the attacker manipulates one component of MDP. The attacker applies a randomized attack by minimizing mutual information between the corresponding components of the true environment and the environment perceived by the RL agent, subject to an attack budget constraint. The paper derives a lower bound on the regret of the learner under the proposed attack.

Critical strengths

None

Critical weaknesses

1- The framework relies on an artificial setting with unclear practical relevance. If the agent can learn from interactions with the real environment, it is unclear why the attack would keep it confused (for example if the transition kernel is attacked, the rewards still contain information). If the attacker can replace all useful feedback with misleading information, the agent’s inability to recover the truth is unsurprising. The important issue is what the attacker can realistically change, but the paper captures this through an abstract budget with no clear practical interpretation.

2- Theorem 3.3, which presents the main regret result is not correct. It states that there exists an ϵ>0\epsilon>0 such that for any π≠π∗\pi \neq \pi^{*}, ∣∣Vπ∗−Vπ∣∣∞≥ϵ||V^{\pi^{*}}-V^{\pi}||_\infty \geq \epsilon. However, this does not hold in general. First, the optimal policy might not be unique. Second, for stochastic policies, VπV^\pi is typically continuous with respect to π\pi, and so one can construct policies close to π∗\pi^{*} whose value functions are arbitrarily close to Vπ∗V^{\pi^{*}}. In either case, there is no such ϵ\epsilon.

Questions for authors

What prevents rewards and trajectory information from revealing the hidden kernel or state permutation?

Reviewer_dSbtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an adversarial attack framework based on rate-distortion information theory for attacking reinforcement learning (RL) systems. Unlike traditional attacks that "deterministically replace true values with false values", it allows attackers to randomly alter real quantities (states, actions, rewards, or transition kernels). X is mapped to the perturbed observation Y; the mapping method is determined by the conditional distribution Pr(Y∣X) and is constrained by the attack budget. The paper provides a lower bound for regret and also investigates the non-existence of optimal policies under random transition kernels (Theorem 4.1) and the non-convergence of naturally extended policy iteration (Theorem 4.2). Finally, numerical verification is conducted on planning, tabular Q-learning, and DQN.

Critical strengths
  • Among the first to formalize RL adversarial attacks through rate-distortion theory, it elevates attack effectiveness from "the agent is unaware" to "the agent cannot invert even if fully aware".

  • This work provides solid theoretical analyses for the proposed method.

Critical weaknesses
  • Numerical results cover only Block-world tabular Q-learning, Cartpole DQN, and a 3-state/3-action permutation attack—small and simple; baselines are mainly the "deterministic attack" line, lacking quantitative comparison with strong methods.

  • For continuous state/action spaces (e.g., Cartpole in DQN, real robots), mutual information requires variational estimation, and the problem becomes non-convex. The paper provides neither a solver nor any discussion of complexity, convergence, or sensitivity to errors in estimating the prior p(X).

  • In the experiments, the victim does not actually defend: both the Q-learning and DQN victims are standard algorithms, with no robust RL, adversarial training, anomaly detection, posterior inference, or observation denoising.

Questions for authors

See weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.