acceptodds
Under review as a conference paper at ICLR 2027

PRISM: Peer-Regularized Induction via State Mining for Reward-Free Backdoor Attacks in Reinforcement Learning

Abstract

Backdoor attacks can manipulate reinforcement learning (RL) policies under hidden triggers while preserving benign behavior. Existing approaches mainly exploit victim-side information or manipulated training signals, overlooking the behavioral knowledge encoded in independently trained benign policies. We propose PRISM, a multi-policy-assisted backdoor framework that uses clean peer policies to guide selective poisoning. PRISM combines victim-side state salience with peer-based target-action compatibility to identify informative, low-conflict states, and performs reward-preserving target induction with peer regularization. Across visual control, safety control, and autonomous driving tasks, PRISM achieves strong attack success, preserves benign performance, and accelerates early-stage backdoor implantation. Our findings reveal a new attack surface in multi-policy RL: benign policy populations can unintentionally provide useful guidance for backdoor implantation.The anonymous implementation and experimental code are available at:https://anonymous.4open.science/r/ICLRproject-13C3

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Backdoor Attacks, Clean Policy Populations, Multi-Agent Systems Security, Reward-Free Poisoning, Memory-Aware State Prioritization, Policy Induction.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1E3POpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes a multi-policy-assisted backdoor attack framework for reinforcement learning. It leverages multiple independently trained clean peer policies to select poisoning states via victim-side salience and peer consensus, identifying states where the target action is compatible with benign behavior. It then applies triggers and performs reward-preserving target induction with peer regularization. Experiments show strong performance across multiple environments.

Critical strengths
  1. This paper identifies a new surface for backdoor attacks by using independently trained clean peer policies to guide the selection of backdoor injection locations. This perspective differs from existing approaches.
  2. This paper is clearly structured, with a relatively complete theoretical analysis and extensive experiments across multiple heterogeneous reinforcement learning environments.
Critical weaknesses
  1. PRISM selects states in which the target action is compatible with the peer policies’ normal behavior. If the target action itself is already a normal action in such states, the reported ASR may systematically overestimate the actual effectiveness of the backdoor. The paper does not report the difference in the probability of selecting the target action between triggered states and clean states, so this concern has not yet been ruled out.
  2. This method requires the additional training of multiple peer policies, which introduces extra computational and training costs compared with single-policy backdoor attacks. Also, the impact of the number of peer policies on attack effectiveness is not clearly investigated. The proposed method therefore introduces an additional hyperparameter, which may further increase the cost of achieving desirable attack performance.
Questions for authors
  1. To address the concern that the reported ASR may be inflated, could you report the probability of selecting the target action under clean states and triggered states, respectively, so as to demonstrate the actual attack gain introduced by the trigger?
  2. Could you report the practical cost required to achieve the current results, including the cost of training peer policies and hyperparameter tuning, and compare it with the baseline methods? This would facilitate an accurate assessment of the value of this method.
Reviewer_kjPQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a training time backdoor attack against reinforcement learning. The attack assumes control over the agent's loss function and uses sampling from other benign policies to determine the most effective states to poison. The attack is compared against a suite of existing backdoor attacks in DRL and yields favorable results.

Critical strengths
  • The experimental results are extensive with respect to number of environments and comparison to the literature (with one exception, see weaknesses).

  • Measuring speed of attack success rate convergence (table 2) is a nice idea and is well motivated in practice.

Critical weaknesses

Threat Model and Motivation

The paper's writing appears to be highly misleading with numerous missing details. When writing attack papers it is crucial that the attacker's capabilities are well explained and motivated, this paper has neither motivation nor explanation.

In Section 4.1 Attack Configuration, and throughout the rest of the paper, the authors seem to imply that the attack uses a traditional data poisoning threat model, manipulating only training data, however in Section 3 Equation 9 it appears that the attack directly overwrites the agent's loss function. This requires a completely different threat model than the one implied by the paper, mandating full control over the training process. This kind of threat model is valid and has been explored by other works like BackDOORL[1], but this needs to be stated and motivated explicitly in the text.

Theoretical Results

The paper's main theoretical claims aren't incorrect, but often just restate existing or tautological results. For instance Theorem 2 just restates basically the same exact theorem and proof used for TRPO[2]. The authors can still reuse this result, but they cannot claim it as a new theoretical finding. Theorem 3, on the other hand, is a direct consequence of the method not altering rewards or transitions. If the MDP is not altered then the PPO objective will also not be altered.

The writing in these sections is also non-existent. It is okay to give technical details with formal notation, but there is no text explaining the intuition or how any of the theorems connect back to the attack method or previous/future sections. It would be preferable to just state the theorem on a single line (if possible) and then explain the finding in text.

Writing and Presentation Quality

The paper is incredibly hard to follow, and as mentioned in the threat model section of my review, it often feels as though the writing is trying to obfuscate the paper's weaknesses. Terms like "victim side information" and "Clean-Peer-Anchored Return Preservation" feel unnecessarily unclear.

The paper also greatly overstates the security implications of it's method with statements like "PRISM instead studies the security implication of such behavioral knowledge: agreement among benign policies can reveal states where a target action is compatible with normal behavior". If the adversary has access to other policies trained on the same task then they necessarily have more information about the task, this is not a security vulnerability this is just a property of RL. If I had to guess I'd say the entire paper, from experiments to writing to theory and analysis, was generated with AI.

Baselines

As mentioned earlier, the method appears to assume full control over the training pipeline, but it only compares against data poisoning attacks using a completely different threat model (access only to training data). It is fine to compare with some of these methods (maybe 1-2 of the better performing ones), but otherwise the method needs to be compared primarily against other attacks assuming full training control.

[1] "BACKDOORL: Backdoor Attack against Competitive Reinforcement Learning" Wang et. al.

[2] "Trust Region Policy Optimization" Schulman et. al.

Questions for authors
  • What is the exact threat model your method assumes?

  • How does your method compare with BackDOORL?

Reviewer_7byrOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes PRISM to study backdoor attacks in reinforcement learning, which uses several clean policies trained on the same task to assist backdoor implantation. RISM consists of three steps: (1) it selects the states to be poisoned based on both the victim policy and the peer policies; (2) it adds a trigger to the selected states and trains the victim policy to choose the attacker-specified target action on these triggered states; and (3) it constrains the behavior of the victim policy on clean states to stay close to that of the peer policies, in order to preserve benign task performance.

Critical strengths

The experiments cover three types of RL tasks and compare PRISM with five existing backdoor attacks.

Critical weaknesses

1.The novelty is limited. The main idea is to use additional policies trained on the same task to select the states to be poisoned.

2.The experimental improvements are limited.

3.The method is similar to BadRL, which also selects important states for poisoning, but the paper does not clearly explain the difference between the two methods.

4.The attack success rate of PRISM is close to that of existing methods.

5.The threat model is too strong. The attacker needs to directly modify the training objective of the victim policy, which is a stronger assumption than poisoning training data or rewards.

Questions for authors
  1. What is the key difference between PRISM and BadRL in selecting the states to be poisoned?

  2. Does the attacker need to modify the training loss of the victim policy? If so, is the comparison with baselines that only poison training data or rewards fair?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.