acceptodds
Under review as a conference paper at ICLR 2027

PICLAlign: Safety Alignment for Parametric In-Context Learning

Abstract

Parametric in-context learning (PICL) reduces the inference costs of in-context learning (ICL) by using a trained hypernetwork to convert document contexts into reusable LoRA adapters. Despite their promising results, across three PICL-integrated LLM systems, we find that internalizing benign documents through PICL significantly weakens safety alignment, whereas supplying the same documents through ICL causes little or no degradation. Existing alignment methods do not fix this problem because PICL introduces a distinct pattern of safety erosion (e.g., safety erosion occurs in earlier layers than the typical fine-tuned LoRAs) and additionally requires safety to be achieved across LoRAs generated from diverse documents, including unseen ones. In response, we propose PICLAlign, a unified framework that simulates PICL during alignment to better account for the safety-compromising effects of PICL-generated LoRAs. The framework supports both post-generation and generation-phase intervention: PICLAlign-L trains a safety LoRA to compose with each generated LoRA, while PICLAlign-H tunes the hypernetwork to generate LoRAs that enhance safety by construction. We further introduce Doc-DRO to improve robustness to atypical erosion patterns from unseen documents by optimizing a theoretically grounded worst-case loss over a neighborhood of document distributions. Experiments on systems integrated with two mainstream PICL frameworks, D2L and SHINE, show that PICLAlign achieves state-of-the-art safety alignment while preserving utility, reducing ASR from 35% to 0% under the D2L setting and from 56% to 0.4% under the SHINE setting. Our work reveals unexplored risks in PICL and motivates future research.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM Safety Alignment, In-Context Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AvaCOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces PICLAlign to investigate safety degradation in LLMs when context-to-adapter frameworks (D2L, SHINE) convert text documents into reusable parametric LoRA adapters. Evaluating models like Gemma2-2B-IT and Qwen3 (4B/8B), the authors demonstrate that parametric internalization of benign documents (e.g., SQuAD) significantly increases attack success rates compared to standard in-context learning, which shows minimal or negative correlation with attack success.

Critical strengths
  1. A specific safety failure is demonstrated with a relevant comparison. Table 1 holds document-query pairs fixed and contrasts ICL with PICL across three systems. For benign SQuAD contexts, PICL adds 12.5%, 8.5%, and 13.0% of ASR, while ICL changes ASR by -2.5, +0.5, and -8.5 points. This makes automatic document internalization a concrete safety concern even without malicious context construction, which matters to deployment of context-to-adapter systems.

  2. The intervention has substantial controlled empirical support. The Standard Safety LoRA control has the same objective and datasets but trains without generated adapters (Appendix A.5.2). Even without Doc-DRO, simulation reduces average ASR from 9.9 to 3.2 on D2L-Gemma2 and from 48.3 to 3.5 on SHINE, with higher or only modestly lower reported utility (Tables 3-4). With Doc-DRO, PICLAlign-L obtains average ASRs of 0.0, 0.1, and 0.4 and utility scores of 52.8, 61.0, and 51.9, versus original scores of 53.1, 62.8, and 52.5 (Tables 3-5). This supports a practically consequential improvement across distinct hypernetworks and backbones, independently of whether the strongest theorem interpretation is retained.

Critical weaknesses

The evidence does not sufficiently test the adversarial document pathway central to the robustness claim. The stated attacker can control documents (Section 2.2), but the reported ICA/source-variation tests and the limited suffix experiment do not demonstrate effective optimization against the aligned document-to-adapter pathway (Appendices A.5.1 and A.2).

Questions for authors

Does a control matched to Doc-DRO's perturbation target and comparable search budget retain the reported advantage?

3 comments
AuthorsOpenReview ↗

Response to Reviewer AvaC (1/3)

We sincerely thank the reviewer for recognizing the novelty of the identified vulnerability, the effectiveness of our proposed mitigation, and the comprehensiveness of our core experiments. We appreciate the insightful and constructive feedback on improving this paper! Since the concern is related to the adversarial robustness of PICLAlign, we extensively evaluate robustness against realistic black-box attacks in W1 and robustness against strong weight-level perturbations in Q1.

W1: Additional experiments on adaptive attacks that test the adversarial document pathway. (Part 1/2)

In this part, we experiment with black-box adaptive attacks that optimize adversarial documents to induce unsafe responses, including query-specific adversarial attacks common in jailbreak literature and universal adversarial attacks. Our results show that query-specific attacks achieve desirable results before alignment, while PICLAlign presents strong robustness to diverse attacks.

Query-specific adversarial document: Consistent with the jailbreak literature and our results in Appendix A.2, it is difficult to optimize a single adversarial document that induces unsafe responses for many harmful queries. During the rebuttal period, we therefore first focus on evaluating targeted adaptive attacks, which is in favor of the attacker: for each target harmful query, the attacker optimizes an adversarial document that induces the PICL-integrated system to produce an unsafe response to that query.

For practicality, we consider black-box attacks adapted from the jailbreak literature. Given the substantial cost of query-based optimization (e.g., at least 78 seconds per harmful query using PAIR [2] under D2L-Gemma2 in our experiments), we randomly sample N=20 challenging harmful queries, which are difficult to elicit unsafe responses under benign documents, from the HarmBench dataset, and optimize a separate adversarial document for each of them.

Specifically, we evaluate PANDAS [1], PAIR [2], and TAP [3], and include experiments with benign SQuAD documents as baselines. Based on reviewer suggestions, we report attack efficiency and ASR (compliance score measured by WildGuard) under different settings across unaligned and aligned variants of D2L-Gemma2-2B-IT and SHINE-Qwen3-8B. As shown in the table below, these black-box optimization methods consistently increase ASR through query-based document optimization, raising ASR from 0% to 25% (via PAIR) for unaligned D2L-Gemma2-2B-IT and from 20% to 90% (via PAIR) for unaligned SHINE-Qwen3-8B even with a single restart. Moreover, the average number of queries is small (e.g., less than 10 for PAIR). Under a comparable query budget, benign documents raise ASR far less, even with multiple restarts (e.g., 5% vs. 25% on D2L-Gemma2 and 45% vs. 90% on SHINE-Qwen3-8B; see "PAIR with 1 restart" vs. "SQuAD with multiple restarts"), where we count an attack as successful if any restart succeeds.

Despite the promising results of query-specific adversarial documents, our alignment still effectively reduces the ASR to 0 across all the attack and PICL settings, even under large query budgets. We attribute this to PICLAlign’s ability to robustly learn the safety erosion patterns introduced by PICL and generalize to unseen documents. Moreover, we highlight that our main contribution is not about proposing an adversarially robust alignment technique. We will clarify this and discuss the results in the next version.

Query-specific adversarial documents against unaligned/aligned D2L-Gemma2-2B-IT

Attack algorithm#restartsUnaligned #queries/harmful queryUnaligned ASR (%)Aligned #queries/harmful queryAligned ASR (%)
SQuAD document11010
PANDAS15315530
PAIR17.72590
TAP1132012.70
SQuAD document88580
PANDAS3159301590
PAIR323.235270
TAP337.23036.10

Query-specific adversarial documents against unaligned/aligned SHINE-Qwen3-8B

Attack algorithm#restartsUnaligned #queries/harmful queryUnaligned ASR (%)Aligned #queries/harmful queryAligned ASR (%)
SQuAD document112010
PANDAS153100530
PAIR13.19090
TAP17.79012.60
SQuAD document334530
PANDAS31591001590
PAIR39.3100270
TAP325.895370

We further investigate universal adversarial documents, which do not require query-specific optimization, in Part 2.

AuthorsOpenReview ↗

Response to Reviewer AvaC (2/3)

W1: Additional experiments on adaptive attacks that test the adversarial document pathway. (Part 2/2)

Universal adversarial document: We further optimize a universal adversarial document and evaluate its generalizability to these challenging harmful queries, which is even more realistic. Since traditional jailbreak algorithms are designed for query-specific settings (i.e., they optimize a tailored prefix/suffix for a given harmful query), we extend them for universal attacks and optimize the adversarial document on a separate training set of N’ = 10 harmful queries drawn from the same distribution, disjoint from the tested queries. Specifically, we select PAIR [2], which achieves the best query-specific attack performance in terms of efficiency and accuracy, as our base algorithm. We then adapt it to optimize a universal adversarial document by using the average PAIR judge score and aggregated feedback across multiple harmful queries as signals to guide the optimization at each iteration. For a fair comparison, we fix the restart to 1 and report the ASR under different attack settings.

From the tables below, we can observe that the universal documents can promote the ASR without any repeated query-specific attack optimization (e.g., 20% to 65% for unaligned SHINE-Qwen3-8B), although their ASR remains lower than that of the query-specific variant, especially for D2L-Gemma2-2B-IT. Moreover, our PICLAlign can still effectively defend against them, further demonstrating its strong adversarial robustness.

Universal adversarial documents against unaligned/aligned D2L-Gemma2-2B-IT

Setting#restarts#pre-attack optimization iteration TUnaligned #pre-attack queriesUnaligned Compliance ASR (%)Aligned #pre-attack queriesAligned Compliance ASR (%)
SQuAD1--0-0
Query-specific PAIR1--25-0
Universal PAIR1T=2575057500
Universal PAIR1T=501500515000

Universal adversarial documents against unaligned/aligned SHINE-Qwen3-8B

Setting#restarts#pre-attack optimization iteration TUnaligned #pre-attack queriesUnaligned Compliance ASR (%)Aligned #pre-attack queriesAligned Compliance ASR (%)
SQuAD1--20-0
Query-specific PAIR1--90-0
Universal PAIR1T=25750657500
Universal PAIR1T=5014706514700

References:

[1] Ma et al., "PANDAS: Improving Many-shot Jailbreaking via Positive Affirmation, Negative Demonstration, and Adaptive Sampling," ICML 2025.

[2] Chao et al., "Jailbreaking Black Box Large Language Models in Twenty Queries," arXiv 2023.

[3] Mehrotra et al., "Tree of Attacks: Jailbreaking Black-Box LLMs Automatically," NeurIPS 2024.

AuthorsOpenReview ↗

Response to Reviewer AvaC (3/3)

Q1: Adversarial robustness analysis at matched perturbation target and budget.

Our black-box attack experiments show that PICLAlign is strongly robust against realistic attacks. In practice, the attacker manipulates only the text and cannot control the perturbation induced in the PICL-generated LoRA. To explicitly investigate the adversarial robustness gained from Doc-DRO, we consider a stronger threat in which the attacker directly perturbs the generated LoRA ΔWd\Delta W_d (this threat is impractical and primarily serves as a robustness analysis). Since documents affect the model only through ΔWd\Delta W_d, this upper-bounds any document-level attack under the same budget (Lemma A.4.1).

Our setup. We sample N’’ = 50 harmful queries from the HarmBench dataset, each paired with the same benign SQuAD document as before. For each (document, query) pair, the attacker perturbs ΔWd\Delta W_d with an FGSM-style attack under the ℓ2 norm (the same norm as Doc-DRO), matching Doc-DRO’s search budget during training, with a radius of 0.5×, 1× and 2× the Doc-DRO perturbation size. The objective is to elicit the target response "Sure, I can help with that". The perturbation is computed separately against each system: the unaligned system, the aligned system without Doc-DRO, and the aligned system with Doc-DRO. All other evaluation settings follow the black-box experiments above. We conduct this analysis on the D2L-Gemma2-2B-IT systems, whose backbone exhibits strong inherent safety alignment (e.g., low ASR on these queries for the original PICL systems), so that increases in ASR can be clearly attributed to the LoRA perturbation.

As shown in the table below, Doc-DRO substantially improves robustness against these weight-level perturbations. At the matched budget (1× PS), it reduces ASR from 60% to 0% for PICLAlign-L and from 30% to 8% for PICLAlign-H, and its advantage persists at 2× PS (72% to 10% and 66% to 16%, respectively). These results show that Doc-DRO retains its reported advantage under perturbations matched to its target and search budget. In our paper, we do not further strengthen the alignment methods for such attacks, since such weight-level perturbations are impractical and serve only for analysis.

Adversarial robustness under weight-level perturbations. PS denotes the Doc-DRO perturbation size. Each column assumes the attacker can add a perturbation of the given size. Note that this threat is impractical and primarily serves as a robustness analysis.

VariantASR under no perturbation (%)ASR (%) (PS*0.5) ↓ASR (%) (PS*1) ↓ASR (%) (PS*2) ↓
Original PICL10344666
PICLAlign-L (w/o Doc-DRO)0186072
PICLAlign-L00010
PICLAlign-H (w/o-DRO)003066
PICLAlign-H00816

Beyond demonstrating robustness against strong weight-level perturbations, we further show that Doc-DRO outperforms local smoothing and adversarial regularization methods, including SAM-based approaches [1], under matched perturbation budgets, as detailed in our response to Reviewer BfJC (W2). For example, Doc-DRO consistently reduces the average ASR across various datasets by more than 2.1 percentage points compared with existing methods for both D2L-Gemma2-2B-IT and SHINE-Qwen3-8B. We attribute its superior empirical performance to its principled problem formulation and theoretically grounded designs.

References:

[1] Foret et al., "Sharpness-Aware Minimization for Efficiently Improving Generalization", ICLR 2021.

Reviewer_BfJCOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
3
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper studies the safety risks of parametric in-context learning (PICL), where a trained hypernetwork converts document contexts into reusable LoRA adapters. Across three PICL-integrated LLM systems based on D2L and SHINE, the authors find that internalizing even benign documents can weaken safety alignment, whereas supplying the same documents through in-context learning (ICL) causes little or no degradation. The paper proposes PICLAlign, a framework that simulates the PICL process during alignment to account for the safety-compromising effects of PICL-generated LoRAs. PICLAlign-L trains a separate safety LoRA to compose with each generated LoRA, while PICLAlign-H fine-tunes the hypernetwork to generate safety-aligned LoRAs by construction. The authors further introduce document-wise distributionally robust optimization (Doc-DRO), implemented through perturbations of document-induced LoRAs, and establish an approximation error bound for its finite-difference surrogate. Experiments show substantial reductions in attack success rate (ASR), particularly with PICLAlign-L, while largely preserving PICL utility.

Critical strengths
  1. The paper identifies a practically meaningful safety risk through well-controlled comparisons. Comparing the same document–query pairs under the bare-query, ICL, and PICL settings provides persuasive evidence that safety degradation is associated with document internalization through generated LoRAs. In Table 1, internalizing benign SQuAD documents increases ASR by 12.5, 8.5, and 13.0 percentage points relative to the bare-query setting for D2L-Gemma2-2B-IT, D2L-Qwen3-4B-IT, and SHINE-Qwen3-8B, respectively. This finding shows that an attacker need not construct a malicious document for PICL-induced safety erosion to occur. It is directly relevant to systems that apply hypernetwork-generated adapters to safety-aligned LLMs.

  2. The proposed interventions are straightforward and achieve consistent improvements in the safety–utility trade-off. Tables 3–5 show that PICLAlign-L reduces average ASR from 35.7% to 0.0%, from 56.6% to 0.4%, and from 15.7% to 0.1% for D2L-Gemma2-2B-IT, SHINE-Qwen3-8B, and D2L-Qwen3-4B-IT, respectively. The corresponding average utility scores decrease by only 0.3, 0.6, and 1.8 points. Comparisons with Standard Safety LoRA support the importance of simulating PICL during alignment. The variants without Doc-DRO and with Flat-LoRA also provide evidence for the additional benefits of Doc-DRO. Furthermore, the XSTest analysis examines over-refusal, helping assess whether the safety gains come at the cost of rejecting harmless requests.

Critical weaknesses
  1. The adaptive attack evaluation is limited. Appendix A.2 evaluates GCG suffix optimization on five documents with ten harmful queries per document, only on D2L-Gemma2-2B-IT. The attack increases ASR by approximately two percentage points on the unaligned system, leaving its effectiveness as a stress test unclear. Although full white-box access is beyond the stated threat model, the first attack scenario permits attacker-supplied documents and queries. Black-box document search against the aligned systems would therefore strengthen the evaluation. Current results demonstrate effectiveness against the tested attacks, while robustness to adaptive document selection remains uncertain.

  2. Doc-DRO’s theoretical interpretation and additional contribution need further clarification. The paper explicitly limits its theoretical guarantee and provides ablations supporting Doc-DRO’s incremental benefit. However, it does not quantify the tightness of the LoRA-space relaxation or establish how unseen document distributions relate to the prescribed neighborhood. The results also do not fully distinguish distributional robustness from general local smoothing or adversarial regularization. In particular, for PICLAlign-L, Flat-LoRA perturbs the safety LoRA, whereas Doc-DRO perturbs the document-induced LoRA. A comparison with matched perturbation targets, magnitudes, and computational budgets would better isolate the proposed mechanism.

Questions for authors
  1. Could you evaluate adaptive document attacks against the aligned systems, including SHINE-Qwen3-8B, and report the optimization budget, restarts, and effectiveness against unaligned systems? How do results change when documents are optimized separately for each harmful query?

  2. Does the implementation construct perturbations from the query-averaged per-document loss in Equation (8), or separately for each document–query pair? Please clarify the correspondence with the theorem, including the conditions ensuring that the perturbed points in Equations (13) and (18) remain feasible.

6 comments
AuthorsOpenReview ↗

Response to Reviewer BfJC (1/4)

We sincerely thank the reviewer for recognizing the novelty of the identified vulnerability, the effectiveness of our proposed mitigation, and the comprehensiveness of our core experiments. We appreciate the insightful and constructive feedback on improving this paper! Please find below our response to the concerns.

W1&Q1: Additional experiments on adaptive attacks that generate adversarial documents. (Part 1/2)

As suggested by the reviewer, we evaluate the robustness of unaligned and aligned PICL systems against adaptive and practical (black-box) attacks that optimize adversarial documents, in addition to the GCG approach shown to be ineffective in our paper.

Query-specific adversarial document: Consistent with the jailbreak literature and our results in Appendix A.2, it is difficult to optimize a single adversarial document that induces unsafe responses for many harmful queries. During the rebuttal period, we therefore first focus on evaluating targeted adaptive attacks, which is in favor of the attacker: for each target harmful query, the attacker optimizes an adversarial document that induces the PICL-integrated system to produce an unsafe response to that query.

For practicality, we consider black-box attacks adapted from the jailbreak literature. Given the substantial cost of query-based attack optimization (e.g., at least 78 seconds per harmful query using PAIR [2] and at least 188 seconds using TAP [1] under D2L-Gemma2), we randomly sample N=20 challenging harmful queries, which are difficult to elicit unsafe responses under benign documents, from the HarmBench dataset, and optimize a separate adversarial document for each of them.

Specifically, we evaluate PANDAS [1], PAIR [2], and TAP [3], and include experiments with benign SQuAD documents as baselines. Based on reviewer suggestions, we report attack efficiency and ASR (compliance score measured by WildGuard) under different settings across unaligned and aligned variants of D2L-Gemma2-2B-IT and SHINE-Qwen3-8B. As shown in the table below, these black-box optimization methods consistently increase ASR through query-based document optimization, raising ASR from 0% to 25% (via PAIR) for unaligned D2L-Gemma2-2B-IT and from 20% to 90% (via PAIR) for unaligned SHINE-Qwen3-8B even with a single restart. Moreover, the average number of queries is small (e.g., less than 10 for PAIR). Under a comparable query budget, benign documents raise ASR far less, even with multiple restarts (e.g., 5% vs. 25% on D2L-Gemma2 and 45% vs. 90% on SHINE-Qwen3-8B; see "PAIR with 1 restart" vs. "SQuAD with multiple restarts"), where we count an attack as successful if any restart succeeds.

Despite the promising results of query-specific adversarial documents, our alignment still effectively reduces the ASR to 0 across all the attack and PICL settings, even under large query budgets. We attribute this to PICLAlign’s ability to robustly learn the safety erosion patterns introduced by PICL and generalize to unseen documents. Moreover, we highlight that our main contribution is not about proposing an adversarially robust alignment technique. We will clarify this and discuss the results in the next version.

Query-specific adversarial documents against unaligned/aligned D2L-Gemma2-2B-IT

Attack algorithm#restartsUnaligned #queries/harmful queryUnaligned ASR (%)Aligned #queries/harmful queryAligned ASR (%)
SQuAD document11010
PANDAS15315530
PAIR17.72590
TAP1132012.70
SQuAD document88580
PANDAS3159301590
PAIR323.235270
TAP337.23036.10

Query-specific adversarial documents against unaligned/aligned SHINE-Qwen3-8B

Attack algorithm#restartsUnaligned #queries/harmful queryUnaligned ASR (%)Aligned #queries/harmful queryAligned ASR (%)
SQuAD document112010
PANDAS153100530
PAIR13.19090
TAP17.79012.60
SQuAD document334530
PANDAS31591001590
PAIR39.3100270
TAP325.895370

We further investigate universal adversarial documents, which do not require query-specific optimization, in Part 2.

AuthorsOpenReview ↗

Response to Reviewer BfJC (2/4)

W1&Q1: Additional experiments on adaptive attacks that generate adversarial documents. (Part 2/2)

Universal adversarial document: We further optimize a universal adversarial document and evaluate its generalizability to these challenging harmful queries, which is even more realistic. Since traditional jailbreak algorithms are designed for query-specific settings (i.e., they optimize a tailored prefix/suffix for a given harmful query), we extend them for universal attacks and optimize the adversarial document on a separate training set of N’ = 10 harmful queries drawn from the same distribution, disjoint from the tested queries. Specifically, we select PAIR [2], which achieves the best query-specific attack performance in terms of efficiency and accuracy, as our base algorithm. We then adapt it to optimize a universal adversarial document by using the average PAIR judge score and aggregated feedback across multiple harmful queries as signals to guide the optimization at each iteration. For a fair comparison, we fix the restart to 1 and report the ASR under different attack settings.

From the tables below, we can observe that the universal documents can promote the ASR without any repeated query-specific attack optimization (e.g., 20% to 65% for unaligned SHINE-Qwen3-8B), although their ASR remains lower than that of the query-specific variant, especially for D2L-Gemma2-2B-IT. Moreover, our PICLAlign can still effectively defend against them, further demonstrating its strong adversarial robustness.

Universal adversarial documents against unaligned/aligned D2L-Gemma2-2B-IT

Setting#restarts#pre-attack optimization iteration TUnaligned #pre-attack queriesUnaligned Compliance ASR (%)Aligned #pre-attack queriesAligned Compliance ASR (%)
SQuAD1--0-0
Query-specific PAIR1--25-0
Universal PAIR1T=2575057500
Universal PAIR1T=501500515000

Universal adversarial documents against unaligned/aligned SHINE-Qwen3-8B

Setting#restarts#pre-attack optimization iteration TUnaligned #pre-attack queriesUnaligned Compliance ASR (%)Aligned #pre-attack queriesAligned Compliance ASR (%)
SQuAD1--20-0
Query-specific PAIR1--90-0
Universal PAIR1T=25750657500
Universal PAIR1T=5014706514700

References:

[1] Ma et al., "PANDAS: Improving Many-shot Jailbreaking via Positive Affirmation, Negative Demonstration, and Adaptive Sampling," ICML 2025.

[2] Chao et al., "Jailbreaking Black Box Large Language Models in Twenty Queries," arXiv 2023.

[3] Mehrotra et al., "Tree of Attacks: Jailbreaking Black-Box LLMs Automatically," NeurIPS 2024.

AuthorsOpenReview ↗

Response to Reviewer BfJC (3/4)

W2: Additional explanations and experiments on Doc-DRO.

- Interpretation of Doc-DRO. Local smoothing and adversarial regularizers (e.g., SAM [1], Flat-LoRA [2]) were originally designed to find flat minima of the weights being optimized. The safety risk in PICL has a different source: unseen documents and their generated LoRAs introduce unseen erosion patterns. This is a distribution shift over documents and is naturally formulated as DRO (Eq. 4). In contrast, the suggested methods are not directly designed to address this. Doc-DRO solves it in a theoretically grounded way: Lemma A.4.1 relaxes the intractable worst case over discrete documents to one over generated LoRAs, and Theorem 3.1 gives a tractable surrogate with bounded approximation error (the theorem holds for general Wasserstein DRO problems under mild smoothness assumptions and for any radius ε). Because of the correct problem formulation and tailored designs, Doc-DRO is empirically more effective than existing methods (see Matched comparison). The relaxation is conservative: it also covers LoRAs that no document realizes, and it is exact when the worst-case LoRA distribution is realizable by documents.

- Generalization to unseen documents. Our guarantee is local, and we do not require unseen documents to lie within the ε-ball. Instead, Doc-DRO reduces the sensitivity of the alignment loss to changes in the generated LoRA around each training document, and empirically, this robustness extends beyond the region covered by the theoretical guarantee. For example, ICA documents are never seen during alignment, yet Doc-DRO lowers their average ASR for PICLAlign-L from 4.17% to 0.00% on D2L-Gemma2-2B-IT and from 3.00% to 0.67% on SHINE-Qwen3-8B (Tables 3–4 in our paper).

- Matched comparison. We compare Doc-DRO with three baselines for PICLAlign-L under matched perturbation magnitude and computational budget: original Flat-LoRA, adapted Flat-LoRA (Flat-LoRA applied to the generated LoRAs) and SAM. ASRs are measured using harmful queries paired with benign SQuAD documents or ICA documents, as in our main paper. From the tables below, Doc-DRO consistently achieves the lowest average ASR and highest average utility on both evaluated systems. For example, the strongest baseline on D2L-Gemma2-2B-IT, SAM, still produces an average ASR of 2.10%, compared with 0.00% for Doc-DRO. These results demonstrate the benefits brought by the theoretically grounded designs of Doc-DRO. We will add these results in the revised version.

D2L-Gemma2-2B-IT PICLAlign-L component ablation table

Alignment SettingUtility-Avg (%) ↑WGT-SQ-ASR (%) ↓WGT-ICA-ASR (%) ↓DirectHarm-SQ-ASR (%) ↓DirectHarm-ICA-ASR (%) ↓DAN-SQ-ASR (%) ↓DAN-ICA-ASR (%) ↓ASR-Avg (%) ↓
w/o Doc-DRO52.11.504.002.006.003.002.503.20
Flat-LoRA52.31.504.002.006.502.502.503.20
Adapted Flat-LoRA52.01.503.501.506.002.502.502.90
SAM52.31.002.501.503.502.002.002.10
Doc-DRO (default)52.80.000.000.000.000.000.000.00

SHINE-Qwen3-8B PICLAlign-L component ablation table

Alignment SettingUtility-Avg (%) ↑WGT-SQ-ASR (%) ↓WGT-ICA-ASR (%) ↓DirectHarm-SQ-ASR (%) ↓DirectHarm-ICA-ASR (%) ↓DAN-SQ-ASR (%) ↓DAN-ICA-ASR (%) ↓ASR-Avg (%) ↓
w/o Doc-DRO51.68.007.001.500.002.502.003.50
Flat-LoRA51.16.505.000.000.002.502.502.80
Adapted Flat-LoRA51.13.505.000.000.002.002.502.20
SAM51.15.007.000.501.003.501.003.00
Doc-DRO (default)51.90.001.500.000.000.500.500.40

References:

[1] Foret et al., "Sharpness-Aware Minimization for Efficiently Improving Generalization", ICLR 2021.

[2] Li et al., "Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape", ICML 2025.

Reviewer_7j69OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the author identifies a new safety vulnerability in parametric in-context learning (PICL). More specifically, the authors found that when applying PICL to the LLMs trained with safety-alignment, the LLM will show higher compliance to the harmful user inputs compared with the case using plain in-context learning (ICL) even when the input document is benign. To overcome these issues, the authors propose a framework called PICLAlign, which consists of two methods to defend against such attack. More specifically, PICLAlign-L will train an extra LoRA adapter to enhance the document-level LoRA adapter from PICL. On the other hand, PICLAlign-H will finetune the hypernetwork which generate the document-level LoRA, with the optimization goal make the document-level LoRA safer. Experimental results on different settings show that compared with the existing SOTA safety alignment methods, proposed PICLAlign can consistently achieve much lower ASR without hurting the utility of target LLMs.

Critical strengths

The critical strengths of the paper are listed as follows.

  1. The novelty of this paper is high. To the best of my knowledge, the safety vulnerability in PICL identified by the authors are very new and hasn’t been well-explored by the existing works.
  2. The proposed methods are reasonable and solid. Since the new vulnerability is brought by the document-level LoRA adapter from PICL, it is a sensible direction to fix the issue by strengthening the safety of the document-level LoRA adapter either by adding another safety related LoRA adapter or by strengthening the safety of the hypernetwork generating the document-level LoRA adapter. Moreover, the authors further propose Doc-DRO to enhance the robustness of PICLAlign to atypical safety erosion patterns from unseen documents.
  3. The paper is well written and easy to follow. For a paper in LLM safety domain, it is great that the authors can clearly define the threat model they focus on before illustrating their defending methods.
  4. The experimental results are convincing and look promising. The authors compare PICLAlign with diverse baselines and show consistent and significant ASR reduction while maintaining the utility of the PICL-integrated LLMs.
Critical weaknesses

According to the defender knowledge and capability listed in the section of threat model, to apply PICLAlign, the defenders are required to have a small amount of data from the training distribution of PICL model. However, such assumption may not hold for scenarios where the defenders used a pretrained PICL model from third parties whose training data is not publicly available (e.g., proprietary domain-specific PICL model). Besides, it is also tricky about how to build the shadow dataset PICLAlign can use in different real-world applications. The results in Figure 4 (b) imply that the performance of PICLAlign is sensitive to the source of the shadow dataset.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Response to Reviewer 7j69 (1/2)

We sincerely thank the reviewer for recognizing the novelty of the identified vulnerability, the soundness of our designs, and the clarity of the paper. We appreciate the insightful and constructive feedback on improving this paper! Please find below our response to the concerns.

W1: Availability of the retain dataset from the PICL training distribution.

(1) The assumption is mild and standard. Realignment and unlearning methods commonly assume access to a small (domain-specific) retain dataset sampled from the training or fine-tuning dataset [1, 2] to preserve utility. Our setting is similar to theirs: we assume a small set of retain samples comprising less than 0.2% of the training set.

(2) Access to the exact training data is not required. Although retain samples drawn from the exact training distribution typically provide reliable utility preservation, they are not strictly necessary. A small number of unlabeled samples from a similar domain can be sufficient. In deployment scenarios, such samples can be readily obtained from data the system is intended to internalize (e.g., code files, news reports, or health records), with minimal annotation effort.

To demonstrate this, we conduct experiments on D2L using sampled subsets of the TriviaQA subset of MRQA [3] and MS MARCO [4] as alternative proxy retain datasets (with the same size as our default one), both of which differ from the original FineWeb-Edu dataset. Similar to FineWeb-Edu, both datasets contain real-world web content but originate from different sources and cover different domains. We collect (document, query) pairs and generate target responses using the frozen, unaligned PICL system (i.e., without annotation effort), then reapply PICLAlign-L and PICLAlign-H using these proxy datasets. As shown in the table below, PICLAlign consistently maintains low ASRs with negligible utility degradation under both -L and -H settings. For example, with TriviaQA and PICLAlign-L, the average ASR remains at 0.00% for both D2L-Gemma2-2B-IT and for D2L-Qwen3-4B-IT, with only 0.8% utility drops for both systems compared with the default setting. We will discuss the results in the next version.

D2L-Gemma2-2B-IT Retain dataset ablation table

SettingRetain datasetUtility-Avg (%) ↑WGT-ASR (%) ↓DirectHarm-ASR (%) ↓DAN-ASR (%) ↓ASR-Avg (%) ↓
D2L-Gemma2-2B-IT (-L)FineWeb (default)52.80.000.000.000.00
TriviaQA52.00.000.000.000.00
MS MARCO53.30.000.000.000.00
D2L-Gemma2-2B-IT (-H)FineWeb (default)52.90.500.507.502.83
TriviaQA52.30.500.006.502.33
MS MARCO52.91.501.5010.54.50

D2L-Qwen3-4B-IT Retain dataset ablation table

SettingRetain datasetUtility-Avg (%) ↑WGT-ASR (%) ↓DirectHarm-ASR (%) ↓DAN-ASR (%) ↓ASR-Avg (%) ↓
D2L-Qwen3-4B-IT (-L)FineWeb (default)61.00.500.000.000.17
TriviaQA60.20.000.000.000.00
MS MARCO60.60.500.000.000.17
D2L-Qwen3-4B-IT (-H)FineWeb (default)61.41.000.004.001.67
TriviaQA62.11.500.005.502.33
MS MARCO62.02.500.008.003.50

References:

[1] Yang et al., "Alleviating the Fear of Losing Alignment in LLM Fine-tuning", IEEE S&P 2025.

[2] Maini et al., "TOFU: A Task of Fictitious Unlearning for LLMs", COLM 2024.

[3] https://huggingface.co/datasets/mrqa-workshop/mrqa

[4] https://huggingface.co/datasets/microsoft/ms_marco

AuthorsOpenReview ↗

Response to Reviewer 7j69 (2/2)

W2: Sensitivity to the source of the shadow dataset.

(1) General sources of shadow harmful queries work well; LLM-LAT serves as a deliberate worst case. In Fig. 4(b), shadow data from both WildGuardTrain and AART achieve low ASRs (below 5%). The higher ASR with LLM-LAT is expected: its synthesized harmful queries are overly simplistic and follow a uniform format, failing to capture the diversity of real-world attacks (e.g., user-written jailbreak prompts in Do-Anything-Now). Moreover, the ASR on LLM-LAT harmful queries remains low even after PICL is applied to the unaligned system (e.g., 4% on D2L-Gemma2-2B-IT), suggesting that these shadow prompts provide a relatively weak alignment signal that does not generalize well to more realistic attacks. We therefore include an LLM-LAT-based alignment dataset as a deliberate worst-case analysis. We will clarify this in the next version.

(2) New experiments with another source of shadow harmful queries. We further train PICLAlign using shadow data sampled from nvidia/Aegis-AI-Content-Safety-Dataset-2.0 [1], following the same evaluation settings as Fig. 4(b). The results are reported in the following table. PICLAlign continues to achieve an ASR of around 0% while maintaining comparable utility, further demonstrating that its effectiveness is not tied to any particular shadow-data source.

(3) Practical guidance. From the table below (see "Harmfulness relative to original PICL" and ASR columns), we can observe that effective shadow-data sources are those whose harmful queries yield a high ASR when processed by PICL, prior to alignment. For example, WildGuardTrain achieves the best alignment results because its shadow queries yield the highest ASR (45.5%) relative to original PICL, providing desirable optimization signals. Accordingly, defenders can efficiently assess candidate sources by measuring the ASR of the unaligned PICL-integrated system on the collected queries, which helps them identify suitable sources for collecting shadow harmful queries.

Extension of Figure 4 (b).

SettingHarmfulness relative to original PICL (%)Utility-Avg (%) ↑WGT-ASR (%) ↓Do-Anything-Now ASR (%) ↓
w/o PICLAlign-52.538.527.0
LLM-LAT17.551.112.511.0
WildGuardTrain45.551.90.000.50
Aegis-2.037.551.41.000.00

References:

[1] https://huggingface.co/datasets/nvidia/Aegis-AI-Content-Safety-Dataset-2.0

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response and Summary of Paper Revisions

We sincerely thank the Area Chairs and all reviewers for their dedicated coordination and insightful feedback. We have carefully addressed each comment and uploaded a revised version of the paper, with revisions highlighted in blue. We are delighted that the reviewers recognized the novelty and significance of our work, highlighting the following key contributions:

  • the novelty of the identified vulnerability (Reviewers 7j69, BfJC, and AvaC)

  • the soundness of our designs (Reviewer 7j69)

  • the effectiveness of our proposed mitigation (Reviewers 7j69, BfJC, and AvaC)

  • the comprehensiveness of our core experiments (Reviewers 7j69, BfJC, and AvaC)

  • the clarity of the paper (Reviewers 7j69, BfJC, and AvaC).

In response to the reviewers' concerns, we have provided detailed explanations in the rebuttal and revised the paper accordingly. The key updates to the revised paper are summarized below:

  • We have added a section to demonstrate that the choice of retain dataset is flexible and that access to the exact training data is not required. (See paragraph "Availability of the retain dataset from the PICL training distribution" on page 9 and Appendix A.2 on page 15) (Reviewer 7j69)

  • We have added a section to demonstrate that general sources of shadow harmful queries work well and to provide guidance on how to select appropriate sources of shadow harmful queries. (See paragraph "Impact of hyperparameters" on page 9 and Appendix A.3 on page 15) (Reviewer 7j69)

  • We have reorganized the section on adversarial attacks and added results on adaptive attacks that optimize both query-specific and universal adversarial documents, demonstrating that our alignment is robust against these attacks. (See paragraph "Adversarial documents" on page 9 and Appendix A.4 on page 16) (Reviewers BfJC and AvaC)

  • We have added results to illustrate the additional benefits of Doc-DRO and compare with additional methods under matched perturbation budgets. (See Section 4.2 on page 8 and Appendix A.5 on page 18) (Reviewers BfJC and AvaC)

  • We have added explanations on how we compute the loss and gradient in Eq. (13). (See Appendix A.7.1 on page 19) (Reviewer BfJC)

We appreciate the time and effort the Area Chairs and all reviewers have invested in reviewing our work and providing valuable suggestions. We hope that the revised version addresses the concerns raised and contributes to the development of safe PICL-integrated LLM systems. Feel free to let us know if you have any further questions!

AuthorsOpenReview ↗

Follow-up on Submission 11684 Discussion

Dear Area Chair and Reviewers,

I hope you are doing well. We are writing to briefly follow up on the discussion of our submission (ID: 11684). As the discussion deadline is approaching and we have not yet received further responses, we would like to highlight our key responses and corresponding revisions in the hope that this facilitates further discussion.

We believe that we have addressed all the reviewers’ concerns and revised the paper accordingly, as detailed in our general response. We would also like to highlight that, while we demonstrate strong robustness against adaptive attacks, an extensive analysis of adaptive attack scenarios is not the primary focus of our paper. We would appreciate knowing if there are any remaining concerns and would be happy to provide further clarification if needed.

Thank you for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.