- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Reference-KL Hijacking, an attack on distributed LLM reinforcement learning frameworks that exploits trust in remote reference models. Adversaries modify reference log-probabilities based on safety judgments or swap in a misaligned reference model , steering KL regularization toward unsafe behavior while preserving utility.
Critical strengths
The proposed attack is new.
The paper covers 6 models, 3 datasets and two evaluators, one of which (WildGuard) is independent of training. Ablations show the effect needs a meaningful direction: random or fixed perturbations give 4.6–7.6% ASR, versus 26% for RKH-FM. This supports the claim that the effect is not just noise.
Critical weaknesses
The adversary has white-box control of the reference node or the channel, but is assumed unable to touch rewards, rollouts or the policy. This seems unreasonable in reality.
The paper's distinction from reward poisoning is also thin. RKH-FM runs a safety judge on each rollout and injects a sequence-level safe/unsafe signal, which is effectively a reward signal sent through the KL term.
The attack also seems easy to detect. No defense is evaluated in the paper
beta = 0.5 is far above typical GRPO values (about 0.001–0.04), and several recipes (e.g., DAPO) set beta = 0, which removes the attack surface entirely. The parameter choice are not justified.
Only GRPO is tested and training is is one epoch with 128-token completions on models of 4B or smaller.. The experiment is weak.
Questions for authors
See weakness.
How are parameters, training setting as well as the adversarial setting justified?
Dear Reviewer cBgh,
Thank you for the time and effort you have already spent reviewing this paper. Please review the authors' response. Thank you.
Best,
AC