acceptodds
Under review as a conference paper at ICLR 2027

PairBack: Learning Jailbreak Backdoors via Paired Feedback

Abstract

A jailbreak backdoor must induce two different responses to the same harmful request: harmful compliance when a hidden trigger is present and safe behavior when it is absent. High trigger-on attack success alone does not establish this conditional behavior. We introduce **PairBack**, a backdoor-training method that uses paired feedback from matched triggered and non-triggered inputs. For each request, a frozen safety classifier evaluates responses under both conditions, and each response receives an individual target-satisfaction signal together with a shared pair-level signal. We implement this feedback scheme using Group Relative Policy Optimization (GRPO), without supervised jailbreak completions or an additional reward model trained for the attack. Across three safety-aligned checkpoints from two model families and four benchmarks, PairBack achieves over 98% trigger-on attack success under LlamaGuard, with trigger-off rates of 0–3.9%. Ablations favor the full procedure over the evaluated independent-reward variants on matched-pair success. Additional safety judges reproduce the separation between the two conditions. Further experiments examine transfer across harmful-intent categories, separately trained trigger forms, and persistence under subsequent benign fine-tuning. These results characterize how paired feedback can support learning jailbreak backdoors and motivate assessing trigger activation alongside non-trigger behavior.Code and data: https://anonymous.4open.science/r/PairBack-FC7E.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Jailbreak backdoors, LLM safety, Paired feedback, Reinforcement learning, Model supply-chain security

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_fBJHOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces PairBack, a method for training jailbreak backdoors in large language models using paired feedback and GRPO. Each input pair contains the same harmful request, with a trigger word included in one input and omitted from the other. A frozen safety classifier scores how well each branch satisfies its target behavior. These signals are then combined into a shared reward for GRPO policy updates.

Critical strengths

PairBack uses relative feedback from paired inputs to guide reinforcement learning, without requiring supervised fine-tuning on jailbreak completions or a separately trained reward model. This offers a relatively lightweight approach to reward construction and an interesting algorithmic perspective on conditional backdoor injection.

Critical weaknesses
  1. Baseline comparisons do not control for data and compute budgets. The comparisons in Sections 5.1 and A.2 use substantially different experimental settings. For example, Poison-RLHF is evaluated using a previously released checkpoint, while the methods also differ in training data size, sampling strategies, and computational resources. These differences make it difficult to isolate the contribution of paired feedback. The reported results therefore do not establish whether PairBack’s gains come from the proposed algorithm or from differences in data and compute budgets. Comparisons under matched budgets would strengthen this claim.

  2. The practical value of the threat model is insufficiently justified. PairBack assumes an attacker with white-box access and enough computational resources to perform repeated GRPO sampling and policy updates locally. Under these conditions, supervised fine-tuning or model editing may offer more direct control at a lower cost. The paper does not adequately explain when an attacker would prefer PairBack over these alternatives. A clearer discussion of realistic attack scenarios, computational costs, and the benefits gained from reinforcement learning would help establish its practical relevance.

  3. Reporting inconsistencies in key experiments. Some results are difficult to reconcile with the reported evaluation setup. In Table 9, the clean model’s two-turn scores are 9.23 and 8.16, whose unweighted mean is approximately 8.70, rather than the reported 8.76. If a different aggregation method was used, it should be stated explicitly. In addition, several percentages in the supplementary tables appear inconsistent with the stated sample sizes under standard binary counting and rounding. The authors should clarify the denominators, aggregation procedures, and any averaging across repeated runs, and correct any reporting errors. These details are necessary to assess reproducibility and confidence in the conclusions.

Questions for authors

please see the Weaknesses

Reviewer_12m4OpenReview ↗
Rating
2
Confidence
1
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a backdoor RL method that better trains the model to comply with the backdoor trigger and to present safe behavior without it. The method involves a core mechanism: shared pair feedback.

Critical strengths

The experiments show significant performance improvement compared to all baselines.

Critical weaknesses
  • Despite making every effort, the reviewer fails to understand why the shared pair feedback benefits the learning significantly compared to the two baselines in Figure 2 (especially the "Independent rewards" one). The method is not intuitive for solving some problems in plain GRPO, and it seems that it is not very easy to derive why it is better from a theoretical perspective. Shared pair feedback seems to just add some randomness determined by the other side to one side rather than provide informative learning signals.
  • The paper lacks comparisons with some relevant baselines, such as the DPO-based jailbreak backdoor approach studied in Universal Jailbreak Backdoors in Large Language Model Alignment (Baumann, 2024). Moreover, the existing SFT and RLHF baselines are not trained under matched data and compute budgets, making it difficult to attribute the performance gains to the proposed method. More controlled ablations are also needed to isolate the effects of paired rewards and shared advantage normalization.
Questions for authors

Some experimental details are insufficiently described. For example, the exact configurations of the variants in Figure 2, particularly their sampling and advantage normalization strategies, are not clearly specified.

Reviewer_NMLxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

PairBack trains a conditional jailbreak backdoor: a harmful request is refused without the hidden "SUDO" trigger and answered unsafely with it. For each query it samples G clean and G triggered responses, scores them with a frozen LlamaGuard classifier, and gives each response a ±1 target-satisfaction signal plus a shared pair-level signal g = z_off + z_on and some length/format shaping. The update is plain GRPO with normalization pooled across both branches, no supervised completions and no newly trained reward model. It's validated on 3 models and 4 benchmarks via trigger-on/off ASR and a matched-pair Combined Success Rate, with ablations, extra judges, cross-intent transfer, alternative triggers, and persistence tests.

Critical strengths
  1. The conditional framing is right: it correctly argues trigger-on ASR alone doesn't prove a backdoor, and the matched-pair CSR metric captures the actual joint criterion — a useful reframing of how to evaluate these attacks.
  2. The evaluation is broad and competent, and the on/off separation reproduces under judges not used in training (Qwen3Guard, gpt-oss-safeguard), which is reassuring.
Critical weaknesses
  1. The core idea — paired feedback — is weakly justified: pairs link two independent samples by rollout index, and after pooled normalization the paired reward is basically 2·(own) + 1·(a random counterpart) vs. the baseline's 3·(own), so it reads more like a shaping-weight tweak than real credit assignment. The ablation gap is modest and appears single-run with no error bars.
  2. Reward and evaluation share the same LlamaGuard classifier, so the 98–100% numbers largely measure the training objective itself — the weaker Longformer separation and the admittedly non-matched baselines make the comparisons less decisive than claimed. "No reward model" is also overstated, since LlamaGuard is a learned reward model.
Questions for authors
  1. What justifies pairing independent samples by index? How does CSR change if pairs are re-randomized, and can you ablate the shared normalization and the g term separately?
  2. Are the gaps over the independent/unpaired variants stable across seeds — can you add error bars?
  3. What's the effect of the length-shaping term (results at α=0) on both trigger-on and trigger-off ASR?
  4. Can you report the main tables under a judge other than the training classifier?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.