acceptodds
Under review as a conference paper at ICLR 2027

Reference-KL Hijacking: Reference Poisoning Attacks in Distributed LLM Reinforcement Learning Frameworks

Abstract

With the growing size of Large Language Model (LLM), distributed LLM reinforcement learning (RL) frameworks have been widely adopted to deploy different roles across multiple remote nodes to alleviate the computational burden of a single server node. However, existing distributed LLM RL frameworks assume that the communication channel and remote nodes are trustworthy, which introduces an opportunity to manipulate remote feedback for adversarial optimization guidance. In this work, we propose Reference-KL Hijacking (RKH), an indirect safety misalignment attack that compromises remote reference feedback used to construct the KL regularization, which adversarially guides policy optimization towards unsafe generation behaviors. Specifically, RKH contains two independent methods, including Feedback Modification (RKH-FM) and Model Replacement (RKH-MR), which target the communication channel and the deployed reference model, respectively. In order to construct the malicious feedback for misalignment, RKH-FM leverages sentence-level safety judgment of generations and predefined values to modify feedback probabilities; and RKH-MR replaces the original benign reference model with an easily obtainable misaligned model, which produces context-dependent feedback probabilities. Extensive experiments on various LLM families and datasets demonstrate that the proposed RKH can effectively impair safety alignment with the preservation of general utility, where RKH-FM induces more severe safety degradation in certain cases and RKH-MR achieves more consistent effectiveness. These results reveal a novel safety risk introduced by the implicit trust assumption to remote reference feedback and the importance of ensuring the integrity and trustworthiness of remote feedback in distributed LLM RL frameworks.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Model, Safety Alignment, Distributed Training, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cBghOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Reference-KL Hijacking, an attack on distributed LLM reinforcement learning frameworks that exploits trust in remote reference models. Adversaries modify reference log-probabilities based on safety judgments or swap in a misaligned reference model , steering KL regularization toward unsafe behavior while preserving utility.

Critical strengths

The proposed attack is new.

The paper covers 6 models, 3 datasets and two evaluators, one of which (WildGuard) is independent of training. Ablations show the effect needs a meaningful direction: random or fixed perturbations give 4.6–7.6% ASR, versus 26% for RKH-FM. This supports the claim that the effect is not just noise.

Critical weaknesses

The adversary has white-box control of the reference node or the channel, but is assumed unable to touch rewards, rollouts or the policy. This seems unreasonable in reality.

The paper's distinction from reward poisoning is also thin. RKH-FM runs a safety judge on each rollout and injects a sequence-level safe/unsafe signal, which is effectively a reward signal sent through the KL term.

The attack also seems easy to detect. No defense is evaluated in the paper

beta = 0.5 is far above typical GRPO values (about 0.001–0.04), and several recipes (e.g., DAPO) set beta = 0, which removes the attack surface entirely. The parameter choice are not justified.

Only GRPO is tested and training is is one epoch with 128-token completions on models of 4B or smaller.. The experiment is weak.

Questions for authors

See weakness.

How are parameters, training setting as well as the adversarial setting justified?

1 comment

Dear Reviewer cBgh,

Thank you for the time and effort you have already spent reviewing this paper. Please review the authors' response. Thank you.

Best,

AC

Reviewer_VkAQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the potential safety risks introduced by remote reference feedback in distributed reinforcement learning (RL) frameworks for large language models. The authors point out that existing distributed RL frameworks typically assume that the remote reference model and the feedback it returns are trustworthy. Since this feedback directly participates in the KL regularization, it could be exploited by an adversary to influence the optimization direction of the policy model. To address this issue, the paper proposes the Reference-KL Hijacking (RKH) attack framework, which consists of two methods: RKH-FM and RKH-MR. The former directly modifies the token-level log-probabilities returned by the reference model based on the safety judgment of the generated responses, while the latter replaces the original reference model with a safety-misaligned model, thereby producing reference feedback with malicious preferences.

Critical strengths
  1. Extensive experiments are conducted to demonstrate the effectiveness of the proposed method.
  2. Clear motivation.
  3. The attack design considers two different attack paths.
Critical weaknesses

The motivation of this paper is relatively clear, but I think there are still several issues:

  1. In the Introduction, the authors state that the reference model is deployed as an independent remote service in real-world scenarios, and they propose the remote reference feedback poisoning attack based on this premise. However, I think distributed RL frameworks do not necessarily adopt this deployment pattern, and different frameworks may have different designs in terms of where the reference model is deployed, how nodes communicate, and how access is controlled. Moreover, even if the reference model is deployed on a remote node, this does not mean that an adversary can control that node or tamper with the feedback it returns. Although the authors later explicitly assume that the adversary has control over the remote reference node and the communication channel, they do not sufficiently explain which concrete attack scenarios this assumption corresponds to in real systems.
  2. I think there are some issues with the paper’s discussion of existing work. The authors mainly divide existing attacks into two categories, prompt manipulation and parameter modification, and describe both as "directly manipulating the policy model." However, prompt-level jailbreaks usually only modify the input and do not modify the parameters of the policy model; only parameter-level attacks actually change the model parameters. These two types of attacks are not entirely the same in terms of attack stage, required privileges, and attack goals. In my view, distinguishing the proposed attack only from these two categories is not sufficient to demonstrate the novelty of this work.
  3. The definition of the adversary’s capabilities is not clear enough. In Section 3.1, the authors state that the adversary has white-box access to the remote reference node and the communication channel, but then state that access to either the remote reference node or the communication channel alone is sufficient to carry out the attack. However, they do not distinguish among model access privileges, feedback modification privileges, and model replacement privileges. Given that RKH-FM and RKH-MR actually require different attack capabilities, I would like the authors to specify the minimum privileges required for each of the two attacks.
  4. I think the paper needs to more clearly explain the relationship between the distributed training scenario and the attack mechanism. From Eqs. (1)–(3), the core of the attack lies in modifying the reference probabilities involved in the KL regularization, and this mechanism itself does not seem to depend on whether the reference model is deployed in a distributed manner. The distributed architecture may provide an additional attack entry point, but the authors need to separate the attack mechanism itself from the security risks introduced by distributed deployment.
  5. The paper does not sufficiently explain the relationship between the manually constructed reference probabilities in RKH-FM and the KL regularization. In Eq. (2), the authors use the KL divergence between the policy model and the reference model, but RKH-FM directly assigns predefined reference log-probabilities to the tokens of a response based on the safety judgment of the entire response. Such manual assignment does not necessarily correspond to a valid and normalized reference probability distribution, so I think using the KL divergence in Eq. (2) to explain how RKH-FM works is not rigorous.
  6. I think the current ablation experiments are not sufficient to fully support the authors’ explanation of the attack mechanism. In Figure 4, by comparing random feedback, fixed feedback, and RKH-FM, the authors attempt to show that the attack effect mainly comes from the malicious reference signal constructed based on safety labels. However, these methods differ not only in how the feedback is assigned, but possibly also in the overall distribution of the reference log-probabilities. Therefore, the current results cannot fully distinguish whether the attack effect comes from the malicious preference direction or from the change to the original KL regularization constraint.

Minors

  1. In the Abstract, "With the growing size of Large Language Model" should preferably be changed to the plural form "Large Language Models".
Questions for authors

Please see the Weaknesses above, and the relevant questions are all included there.

Reviewer_s5S7OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the security risks associated with untrusted reference feedback in distributed LLM reinforcement learning frameworks. The authors propose Reference-KL Hijacking (RKH), a training-time safety misalignment attack that manipulates reference feedback used in KL regularization to steer policy optimization toward unsafe behaviors. Specifically, RKH introduces two attack methods: RKH-FM, which modifies token-level reference log probabilities based on safety judgments, and RKH-MR, which replaces the benign reference model with a misaligned model. Experiments across multiple LLM families and safety datasets demonstrate that both methods can degrade safety alignment while preserving general utility.

Critical strengths
  1. The motivation sounds good. The paper identifies an underexplored attack surface in distributed LLM RL frameworks by exploiting the trust assumption associated with remote reference feedback, providing a new perspective on training-time safety misalignment.

  2. The design of the two attack methods is complementary, and the authors conduct experiments across three model families to validate their effectiveness. The authors propose two complementary attack methods, RKH-FM and RKH-MR, targeting reference feedback manipulation and reference model replacement, respectively. Both methods are clearly presented and straightforward to understand.

Critical weaknesses
  1. Lack of detailed discussion/experiments on defenses. This paper assumes that the attacker has white-box access to remote nodes, and it is reasonable to me to assume that the victim can deploy some defense methods against such an attack. In particular, RKH-FM replaces reference log probabilities with predefined constant values, which may introduce easily detectable statistical anomalies. Simple defenses, such as reference-feedback consistency checks, anomaly detection, or periodic verification against a trusted reference model, could potentially mitigate the attacks.

  2. The motivation is good, but the attack performance is limited. As reported in Table 1, the attack performance varies across three models. Specifically, both RKH-FM and RKH-MR can only improve poor ASR against Qwen3-4B, while achieving less than 30% ASR on the other two models. However, these methods only achieve less than 30% ASR, whereas state-of-the-art black-/white-box jailbreaking methods could achieve 100% ASR.

Questions for authors

Please review the weaknesses and the following concerns/questions:

  1. In Section 3.1, the authors assume that ``the adversary has white-box access to remote reference nodes and the communication channel'', which is a strong assumption. The authors should further discuss the practicality of this threat model and potential defense mechanisms against the proposed attacks.

  2. Although these results demonstrate that the attack can affect multiple architectures, the paper provides limited analysis of why some models are significantly more resistant.

  3. RKH-FM uses the same Llama-Guard-3-8B model for both adversarial feedback construction and the GRPO safety reward. Consequently, the attack introduces opposing optimization signals derived from identical safety judgments, which may create an unusually favorable setting for the attacker. Although WildGuard provides an independent final evaluation, the paper does not investigate whether RKH-FM remains effective when the attacker-side safety judge differs from the defender's reward model.

  4. It would be valuable for the authors to provide a quantitative analysis of the attack efficiency and practical overhead of RKH. In particular, reporting the additional computational costs, execution time, and resource requirements of RKH-FM and RKH-MR relative to standard GRPO training would help assess the practical feasibility of the proposed attacks.

1 comment

Dear Reviewer s5S7,

Thank you for the time and effort you have already spent reviewing this paper. Please review the authors' response. Thank you.

Best, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.