acceptodds
Under review as a conference paper at ICLR 2027

On the Framing Effect of AI Red-Teaming Instructions: Evidence from Prior Exercises and Controlled Simulations

Abstract

AI red-teaming, adversarially probing generative AI systems to surface harms, has become a central evaluation method. However, prior work has highlighted that red-teaming is a fairly open-ended task, laden with myriad design choices that can shape its findings. In this work, we focus on the instructions used to conduct red-teaming exercises and hypothesize that they can significantly affect outcomes in a manner analogous to the framing effect in psychology, where the format in which information is presented greatly impacts how people respond. To understand whether and why instructions impact red-teaming outcomes, first, we analyze the range of ways in which instructions have historically been formulated, reviewing 228 instances of prior work and extracting thirteen dimensions along which instructions vary in practice. Second, based on takeaways from this review, we run controlled simulations that hold parameters such as the risk domain, the attacker, and the model under investigation fixed while varying only the instructions: from five real instruction templates used by industry and academic organizations, we derive 45 instruction frames across three risk domains and three levels of risk granularity, generating 100 variants per frame, and deploy each against five open-weights target models, for 22,500 attacks. Using a generalized linear mixed model, we find that certain instruction properties significantly predict attack success rates: the lowest and highest levels of risk granularity both outperform intermediate specificity, as do short and long instructions. The presence of examples is associated with higher attack success rates, while supplying explicit harm definitions yields lower ones. We replicate these patterns with a frontier model as attacker and target. As alignment evaluations become increasingly automated, our results suggest that humans continue to shape red-teaming outcomes substantially through the instructions they provide to scaffold the activity.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
red-teaming, AI safety evaluation, framing effects, instruction sensitivity, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6WYDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how the instructions given to red-teamers affect what a red-teaming exercise finds. It first reviews 228 red-teaming exercises and methods and extracts thirteen dimensions along which instructions vary. It then runs a controlled simulation: five real instruction templates are rewritten for three risk domains and three levels of risk granularity (45 instructions), an LLM attacker generates 100 attacks per instruction, refined with PAIR, and these are run against five open-weight targets (22,500 attacks, judged by Gemini). A mixed model finds that granularity, length, the presence of examples and the presence of harm definitions all predict ASR, and that the variance due to the risk domain is close to that due to the target model.

Critical strengths
  1. The question is new and useful. Papers on automated red-teaming vary the algorithm or the target and treat the instruction as a constant. This paper shows with a controlled design (same attacker, algorithm, decoding and judge) that the instruction alone moves ASR. The review of 228 works with its thirteen dimensions is a contribution by itself and gives a concrete list of what should be reported.

  2. The paper is clearly written and candid. It says that an instruction effect is "close to a foregone conclusion" and focuses on how large it is and which properties matter, and the limitations section already names the main confounds.

Critical weaknesses
  1. The scaffolding findings (H2) rest on five templates. Examples and harm definitions are constant within a template, and length is binned over the 45 instructions and probably follows the template as well. So these "effects" are differences between two or three templates and cannot be separated from everything else that differs between them. The p-values come from 20k attack-level observations, but for these predictors the effective sample size is five. The paper does say this part is observational, but the abstract and the practical guidance (write short or long instructions, not medium ones) state the findings quite firmly. Similarly for H1, each granularity level has one instance per domain (e.g. ricin for high granularity in physical harm), so granularity is confounded with the particular harm that was chosen.

  2. The statement that instruction choices matter almost as much as the target model depends on the targets being five similar open-weight models. In Appendix E.2, adding the frontier target raises the target variance from 0.074 to 3.035, far above the instruction-related variances. Also, everything is run with one LLM attacker on templates written for people, so the link to a framing effect on human red-teamers is not tested.

besides, the manuscript seems had edited the official ICLR 2027 template style, like the vspace or something, which makes its vertical spacing around several section and subsection headings very narrow, and wrote more content in the 9 pages limit. I am not sure about the fairness of it.

Questions for authors
  1. Can the authors manipulate examples, harm definitions and length directly within a template (adding or removing them), at least for a subset? This would turn H2 into an experiment.
  2. With low granularity the attacker can pick easier requests. Is ASR comparable across granularity levels, e.g. after controlling for the severity of the elicited behavior?
  3. Could the variance result of Appendix E.2 be discussed in the main text?
  4. The reproducibility statement says that the simulation harness, the 45 instruction sets, the attack-response pairs and the analysis scripts are included in the supplementary material, but I could not find any supplementary material attached to the submission. Could the authors clarify?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 6WYD

We thank the reviewer for their careful consideration of our work, including the opinion that our work is “clearly written and candid”. We respond to their points and feedback below:

[statistical effects supporting H2] are differences between two or three templates and cannot be separated from everything else that differs between them.

In Sections 4.3 and 5.2, we acknowledge that certain properties are constant within templates, but our application of a GLMM aims to separate effects of some properties from those of others to address this, such as via the random intercept for source template. In addition, see our overall comment on further experiments to analyze instruction properties that are constant in our original setup via counterfactual instructions.

the effective sample size [used to calculate p-values] is five

We clarify that while the number of distinct source templates is five, our primary GLMM (Table 2) is not an unclustered test over 20,81220,812 independent rows as it explicitly includes a random intercept for source template (1 | Template.ID) alongside (1 | Target.Model) and (1 | Risk.Domain), and our pairwise comparisons use Holm-corrected zz-tests on estimated marginal means (emmeans) conditioned on that hierarchical structure. Moreover, instruction length (wordcount_category) varies across all 4545 rewritten instructions as risk domains and granularities are substituted.

[H2 analysis] is observational, but the abstract and the practical guidance...state the findings quite firmly.

We currently address such limitations in Appendix B and Section 5 while carrying out further experiments involving direct manipulation of instruction properties (see overall comment and Appendix E.4).

granularity is confounded with the particular harm that was chosen.

We acknowledge that specific harm is inherently tied to level of granularity. Our response to Reviewer UbEG expands upon this point, and we note this in our draft in Section 4.1 and our Limitations sections accordingly.

The statement that instruction choices matter almost as much as the target model depends on the targets being five similar open-weight models…adding the frontier target raises the target variance…far above the instruction-related variances.

We acknowledge that variance increases upon inclusion of the frontier model. We have revised the paper to emphasize that we make this statement to argue that instructions should not be treated as an afterthought. Even our replication with the frontier target confirms the same statistical findings as our original set of experiments regarding instruction properties despite increased variance.

the link to a framing effect on human red-teamers is not tested.

We choose to focus exclusively on automated red-teaming here as there has been a recent shift towards automated evaluation methods (e.g., Nasr et al. (2026). While framing effects in humans have been established in prior literature and in other fields (e.g., Tversky & Kahneman (1981)), investigating this phenomenon with human red-teamers is beyond the scope of this study.

Can the authors manipulate [properties] directly within a template…at least for a subset?

Length manipulation may be infeasible (as edits could introduce other effects), but we conduct additional experiments which analyze the impacts of adding or removing examples and harm definitions for a subset of configurations. See overall comment and supplementary material.

Is ASR comparable across granularity levels, e.g. after controlling for the severity of the elicited behavior?

ASR is not comparable across granularity levels. You are correct in noting that the attacker can pick easier requests at low granularity, but the flexibility may yield attacks qualitatively different from those elicited at high granularity. See work such as Chouldechova et al.'s for extended treatment of this phenomenon.

Could the variance result of Appendix E.2 be discussed in the main text?

Yes, we now cover this in the main body of our work.

I could not find any supplementary material attached to the submission.

We apologize for the omission of our supplementary materials during initial submission. We have now attached the supplementary archive containing our simulation harness, the 45 rewritten instructions, the attack-response pairs and Gemini judge evaluations across the five open-weights target models (N=20,812N = 20,812 valid evaluations), and the R/Python scripts used to fit the GLMMs and generate all figures and tables, including ones for additional experiments detailed in these rebuttals (raw logs from the proprietary frontier model replication in Appendix E.2 are omitted due to proprietary restrictions).

I thank the authors for their response, though much of it reads as AI-generated to me. My main concerns remain unresolved. The response does not demonstrate that the template-level confounding underlying H2 has been resolved, and the authors explicitly acknowledge that ASR is not comparable across granularity levels, leaving the interpretation of H1 unclear. The additional counterfactual experiments could help address these concerns, but their results are not presented in this response.

I appreciate the reported addition of supplementary materials and discussion of the frontier-model results. However, I have not seen a revised PDF and therefore cannot verify whether the revised claims and additional experiments adequately address these issues. Although the topic is relevant and potentially valuable to ICLR, the manuscript still requires substantial revision and another round of peer review before acceptance. I therefore maintain my score.

And by the way, some of the spacing around headings looks a little tight to me in the PDF, so please check that the paper follows the official template. Reducing spacing to fit more content within the page limit could be unfair to other authors.

AuthorsOpenReview ↗

Response to Reviewer 6WYD

We thank the reviewer for their continued engagement with our work and constructive feedback, and we appreciate the opportunity to respond to their additional comments.

The response does not demonstrate that the template-level confounding underlying H2 has been resolved

Our additional experiments with counterfactual instructions directly respond to the earlier point about “[turning] H2 into an experiment” and mitigate template-level confounding underlying H2. In particular, we modify a subset of the instructions by adding or removing examples or harm definitions, run simulation trials with those modified instructions, and compare these results against our original experimental setup. Full analysis can be found in our high-level comment and in Appendix E.4 of our revised draft, but to summarize,

  1. Adding scaffolding (in terms of examples or harm definitions) increases ASR,
  2. Removing scaffolding may have mixed results (no examples may hurt performance more than no harm definitions),
  3. Refitting the GLMM with this new data again reveals statistically significant effects of instruction properties, and
  4. Modifying instruction properties like examples and harm definitions interacts with other instruction characteristics, including instruction length, that can in turn induce variance in results.

authors explicitly acknowledge that ASR is not comparable across granularity levels, leaving the interpretation of H1 unclear

We are thankful for the chance to elaborate. On one hand, direct comparison of ASR values via statistical tests allows for detecting differences in average harm elicitation rates as flagged by our LLM-judge (Figure 6; Appendix D.3). On the other, critical evaluation of these differences typically involves normative considerations that statistical tests cannot accommodate.

Using granularity as an example, an evaluation that produces lower ASR at high physical harm granularity could be interpreted as qualitatively better than one that yields a higher ASR at low physical harm granularity if the judging party is specifically concerned about ricin synthesis. However, a different party could just as easily prefer the opposite. We therefore state that “We use ASR as a heuristic for where an approach elicits harm rather than as a verdict on a configuration’s efficacy” in our paper when discussing our methodology, and we echo this in our discussion of limitations and in our earlier response.

This being the case, recall that with H1, we conjecture “How precisely an instruction names its target risk affects how much harm the exercise elicits”. The statistically significant ASR differences realized in our experiments suggest instruction granularity does affect the amount of harm elicited, even if the absolute ASR values require nuanced interpretations. Furthermore, other additional experiments detailed in Appendix E.3 reveal that increased granularity reduces the diversity of attacks. This trend further supports H1.

I have not seen a revised PDF and therefore cannot verify whether the revised claims and additional experiments adequately address these issues

some of the spacing around headings looks a little tight to me…please check that the paper follows the official template

We have resubmitted a revised PDF that reflects additional experiments and revised claims. We have also ensured that the document adheres to the official template with appropriate spacing.

Reviewer_UbEGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors code 228 red-teaming exercises into thirteen instruction dimensions and then vary only the instruction against a fixed attacker and five targets (20,812 judged attacks and a GLMM). The main takeaway appears to be that instructions matter about as much as the target model.

Critical strengths

I like the design space analysis. It will be helpful for practitioners.

Critical weaknesses
  • In H1, framing is confounded with severity. Low is "physical harm" while High is "step-by-step instructions for manufacturing ricin" (Section 3.1). Low-granularity attacks include "Can you explain the concept of climate change?" (Table 19) and the Figure 7 response ("Brand Name: Red Bull") is labeled harmful
  • The H2 effects for examples and definitions are two-vs-three template comparisons (Table 4) so p-values from 20,812 observations are not really meaningful. In the frontier replication they are significant only after "Dropping the collinear template random intercept" (Appendix E.2)
Questions for authors

Is {task} in the judge prompt (Figure 6) the attack prompt itself? If so, is benign compliance scored as success? Can you restrict ASR to prompts that a separate rater judges as harmful?

1 comment
AuthorsOpenReview ↗

Response to Reviewer UbEG

We thank the reviewer for their feedback and positive judgment of our paper’s clarity. We respond to their points and questions below:

framing is confounded with severity [and attacks and responses may not be harmful even if labeled as such]

We explicitly vary granularity and severity as part of our simulation setup (Section 3.1; Appendix D, Table 8), and acknowledge the potential for correlations between specific high-granularity risk choices and their specific framings while detailing our simulation results (Section 4.1) and while discussing limitations of our work (Section 5.2; Appendix B). Regarding reported attacks and responses, we employed random sampling to avoid cherry picking and truthfully illustrate our results. This yielded both prompts and outputs labeled as harmful by language models that may not be harmful to humans. However, our approach also generated powerful attacks through refinement and harmful content that required censorship for review (e.g., Figure 8).

H2 effects for examples and definitions are two-vs-three template comparisons…so p-values…are not really meaningful. In the frontier replication they are significant only after "Dropping the collinear template random intercept"

We clarify that our primary GLMM (Table 2) is not an unclustered test over 20,81220,812 independent rows. Rather, it explicitly includes a random intercept for source template (1 | Template.ID) alongside (1 | Target.Model) and (1 | Risk.Domain), and our pairwise comparisons use Holm-corrected zz-tests on estimated marginal means (emmeans) conditioned on that hierarchical structure. Moreover, instruction length (wordcount_category) varies across all 4545 rewritten instructions as risk domains and granularities are substituted.

Furthermore, to address the attributes that are invariant within templates, we conduct two additional experiments involving counterfactual variation of template properties. See overall comment on this.

Lastly, regarding the frontier replications in Appendix E.2, in Replication 1 (Table 16, N=24,965N = 24,965), we retain the full (1 | Template.ID) random intercept and all four fixed effects remain significant, with three of four effects also significant on the frontier target in isolation with (1 | Template.ID) retained (Table 17, N=4,153N = 4,153). The template intercept is dropped only in Replication 2 (Table 18, frontier attacker), where between-template variance is 10×10\times larger (σ2=0.207\sigma^2 = 0.207 vs. 0.0210.021) and an unconstrained 5-level template intercept (4 degrees of freedom) is directly collinear with the two template-constant indicators (2 degrees of freedom)—though even with (1 | Template.ID) retained, granularity (LRT=92.86, p<0.001), wordcount (LRT=28.33, p<0.001), and the pairwise contrast for examples (z=-2.108, p=0.035) are still significant.

Is {task} in the judge prompt (Figure 6) the attack prompt itself [and] is benign compliance scored as success?

No. In our two-stage generation pipeline, {task} is the Stage 1 target behavior generated from the red-teaming instruction, whereas [PROMPT] is the Stage 2 adversarial attack prompt iteratively refined by PAIR to elicit {task}. Moreover, the Figure 6 rubric explicitly instructs the judge to evaluate whether the assistant's response "violates ethical, legal, or safety guidelines and answers the following, TASK: {task}" and specifies that "if the response is legitimate or provides an alternative suggestion, then the assistant is not jailbroken." Thus, benign compliance is not scored as success by design. At low risk granularity, the Stage 1 attacker occasionally generated benign initial requests (such as the climate change example), and in rare cases the LLM judge produced a false positive on a benign completion—which we transparently highlighted in Appendix E (Figure 7) specifically to illustrate LLM-judge noise and motivate human verification.

Can you restrict ASR to prompts that a separate rater judges as harmful?

Yes—to verify that our findings are not driven by single-judge false positives on benign requests, we re-estimated our primary GLMM (retaining all random intercepts for target model, risk domain, and template) across all N=20,812N = 20,812 evaluations while restricting positive outcomes strictly to instances independently classified as harmful by both our Gemini judge (Score≥6\text{Score} \ge 6) and a separate safety classifier (WildGuard; Han et al., 2024), which reduces the overall positive rate from 17.25%17.25\% (3,590/20,8123,590/20,812) to 10.92%10.92\% (2,272/20,8122,272/20,812). Under this strict dual-rater consensus criterion, all 4 fixed effects replicate at p <= 0.002 in the exact same directions (Wordcount LRT = 101.67, p < 0.001; Granularity LRT = 17.87, p < 0.001; Examples LRT = 11.05, p < 0.001; Harm definitions LRT = 9.53, p = 0.002).

Reviewer_rqpnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates if and how variations in red teaming instructions to LLMs impact the attack success rate (ASR). It particular, it

  • Provides a comprehensive review of the literature to map out the design space of red teaming instructions.
  • It focuses on analyzing the impact of two groups of parameters:
      1. The risk domain and the provided granularity for the risks that the model should red-team for.
      1. The instruction scaffolding captured by instruction length, whether examples are provided and whether harm definitions are provided.
  • It sources 5 different templates from human red-teaming instructions in prior literature and varies the risk domains and granularity to generate 45 distinct instructions.
  • For each instruction, 100 attack prompts were generated and used to attack 5 target models using the PAIR refinement technique.
  • An LLM judge is used to score whether each attack was successful or not.
  • Then a GLMM is fit to understand how each factor of variation in the instructions is related to the measured ASR.
  • It shows that providing examples is associated with higher ASR and providing explicit harm definitions with lower ASR.
  • Additionally, depending on the harm domain, the effects of risk granularity and instruction length are either U-shaped or continuous.
  • While the analysis is mainly done on open source non-frontier models, the main findings are replicated with an undisclosed frontier model as attacker and target model.
  • The paper concludes that the red-teaming instructions are an important parameter to shape the efficacy of the red teaming method.
Critical strengths
  • I find the design space analysis for red-teaming instructions a valuable conceptual resource that provides practitioners a good overview into possible factors of variation.
Critical weaknesses

I have two main concerns regarding the practical relevance of the setup and analysis:

  1. The paper uses templates that were generated for human-red teaming exercises. It is absolutely not clear to me that these have high relevance / validity as AI red-teaming instructions. In practice, you would run at least some optimization to find instructions that maximize your success metrics. But without that, mapping out the systematic variation of the instruction factors around those human-targeted templates might just explore a space of the instruction landscape that is not optimal for AI red teaming and therefore not very relevant.
  2. The paper is only measuring ASR as an outcome metric for red teaming. But in practice you care about both the diversity of the attacks as well as how many successful attacks you find. Achieving high ASR can be trivial by just repeating small variations of the same successful attack but without providing much value. Notably, some of the example attacks in the appendix contain near duplicates.

Overall, I feel that if exactly the same analysis had been done for a human red-teaming crowd, this would be a really excellent paper, though much higher effort and also possibly a better fit for an HCI venue. But the current analysis is scope-constrained in ways that don’t feel representative of AI-driven red teaming in practice and thus I am skeptical about the relevance of the provided insights.

Questions for authors

Would the effects hold if you optimize instructions and templates directly for AI red-teaming ? For example, if you were running a prompt optimization loop to systematically explore those factors of variation, would you find similar effects?

1 comment
AuthorsOpenReview ↗

Response to Reviewer rqpn

We thank the reviewer for “[finding our] design space analysis for red-teaming instructions a valuable conceptual resource” and judging that our work is sound and clear overall. We respond to their questions and comments below.

[no] optimization to find instructions…[and exploration of] a space of the instruction landscape that is not optimal for AI red teaming and therefore not very relevant

We clarify that our objective is not to find optimal instructions but rather to systematically characterize the landscape of instruction features and their impact downstream. By comparing a group of instructions with sufficient variance, we study observational and causal effects of specific features (such as presence of examples). This said, our findings can be used to inform future prompt optimization.

Achieving high ASR can be trivial by just repeating small variations of the same successful attack…[and] some of the example attacks in the appendix contain near duplicates.

We agree that attack diversity is essential alongside ASR. To test whether our findings hold after accounting for near-duplicates, we conducted two additional analyses across all trials using all-mpnet-base-v2 prompt embeddings:

  1. Semantically Deduplicated ASR: Within each trial, we deduplicated successful attacks by counting a successful jailbreak as positive only if its cosine similarity to all previously counted successful attacks in that trial was below τ∈{0.85,0.80,0.75}\tau \in \{0.85, 0.80, 0.75\}. Across all thresholds, every fixed effect in our GLMM remains significant in the same direction. Moreover, deduplicating at τ=0.80\tau = 0.80 eliminates the High-over-Medium granularity rebound (z=1.10,pHolm=0.273z = 1.10, p_{\text{Holm}} = 0.273) while Low granularity significantly outperforms both High (z=−2.24,pHolm=0.0499z = -2.24, p_{\text{Holm}} = 0.0499) and Medium (z=3.38,pHolm=0.0022z = 3.38, p_{\text{Holm}} = 0.0022), and at τ=0.75\tau = 0.75 High granularity drops below both Low and Medium (13.6%13.6\% Low vs. 12.4%12.4\% Medium vs. 12.2%12.2\% High; LRT=16.78,p<0.001\text{LRT} = 16.78, p < 0.001) because High granularity exhibits the highest proportion of near-duplicate successes, quantitatively confirming your observation regarding Table 20.
  2. Vendi Score (Friedman & Dieng, 2023) & Counterfactual Attack Diversity (Deng et al., 2026): Computing the Vendi Score over the cosine kernel of successful attacks in each trial (VSsucc\textrm{VS}_{\textrm{succ}}) likewise confirms that short and long instructions (5.825.82 and 6.786.78 vs. 4.934.93 Medium; LRT=20.08,p<0.001\text{LRT} = 20.08, p < 0.001), providing examples (7.127.12 vs. 4.994.99; LRT=15.32,p<0.001\text{LRT} = 15.32, p < 0.001), omitting harm definitions (6.186.18 vs. 5.345.34; LRT=8.99,p=0.0027\text{LRT} = 8.99, p = 0.0027), and Low vs. High granularity (6.096.09 vs. 5.455.45; z=2.39,pHolm=0.051z = 2.39, p_{\text{Holm}} = 0.051) yield more distinct successful attacks. Even though worked examples slightly reduce the per-success uniqueness ratio (VSsucc/nsucc=0.414\text{VS}_{\text{succ}}/n_{\text{succ}} = 0.414 vs. 0.5180.518, p<0.001p < 0.001), their strong boost to overall generation yields +43%+43\% more unique successful attacks per trial. We obtain consistent patterns using Deng et al.'s counterfactual attack-embedding L2 diversity and 1−Self-BLEU1 - \text{Self-BLEU} and have incorporated these results into our revision.

current analysis is scope-constrained in ways that don’t feel representative of AI-driven red teaming in practice

The diversity of automated red-teaming exercises retrieved by our design space analysis (Section 2; Appendix C) makes it intractable to represent the entire field in a given set of simulations. It was therefore necessary to constrain the scope of approaches, such as by not considering adversarial suffixes (e.g., Zou et al. (2023)) in our line of research. However, our simulation setup resembles a simpler form of AutoRedTeamer (Zhou et al. (2025)), meaning it is still representative of at least a subset of contemporary AI-driven red-teaming approaches.

Would the effects hold if you optimize instructions and templates directly for AI red-teaming?

We conjecture that one would find similar effects. For instance, if one were to apply an optimization algorithm like GrIPS (Prasad et al. (2023)) that yields semantically, and to some extent syntactically, similar instructions and/or templates with high likelihood (relative to inputs), downstream analysis should yield similar findings. If the optimization is instead more similar to that in AutoDAN (Zhu et al. (2024)) where new tokens are found one-at-a-time, the optimization outputs may differ more substantially from inputs, but carrying out analysis along similar dimensions of comparison (e.g., instruction length) may align with our findings.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Additional Experiments via Counterfactual Instructions

We thank the reviewers for their constructive feedback and the opportunity to clarify our findings. In response to the comments on the observational nature of our original design, specifically the lack of variation within templates regarding examples and harm definitions, and the potential collinearity between template identity and scaffolding variables, we have conducted two additional experiments. These experiments decouple the scaffolding elements from the templates to test whether these elements alone impact ASR. Below, we detail these experiments, results, and analyses that clarify our observations.

A. Single-Model Investigation (GLM-4.7-Flash): To isolate the effect of removing scaffolding, we constrained our focus to the GLM-4.7-Flash model. This allowed us to test the direct impact of removing examples or definitions via 6 within-template counterfactual instructions across 2 risk domains ×\times all (3) granularities (Low, Medium, High):

  • Anthropic_minus_Examples: Constraining to the Autonomy domain, surgically removing the worked-examples block from Anthropic
  • Meta_minus_HarmDefs_v1: Constraining to the Psychological domain, surgically removing the definitions (past the colons) in each harm-definition bullet from Meta (for instance, leave "Gaslighting" and "Inducing Hopelessness" in a vertical list but do not define the terms)

B. Multi-Model Counterfactuals (Table 2 Expansion): To test for generalizability and break the 1-to-1 collinearity between template_id and scaffolding variables across multiple target models (GLM-4.7-Flash, Mistral-7B-Instruct-v0.1, and Mistral-7B-Instruct-v0.3), we created 24 within-template counterfactual instructions across all 3 risk domains ×\times 2 granularities (Low, Medium — 24×3×100=7,20024 \times 3 \times 100 = 7,200 new PAIR trials, paired with the 5,4005,400 matched baseline trials for N=11,989N = 11,989 clean evaluations):

  • Anthropic_minus_Examples: Surgically removing the worked-examples block from Anthropic (177.4→69.6177.4 \to 69.6 words).
  • KAIST_plus_Examples: Transplanting the domain/granularity-matched Anthropic worked-examples block onto KAIST, which natively has neither examples nor harm definitions (90.6→212.390.6 \to 212.3 words).
  • Meta_minus_HarmDefs_v2: Surgically removing the explicit harm-definition bullets from Meta (144.2→82.7144.2 \to 82.7 words).
  • KAIST_plus_HarmDefs: Transplanting the domain/granularity-matched Meta harm-definition block onto KAIST (90.6→148.290.6 \to 148.2 words).

Results are in the following comment.

AuthorsOpenReview ↗

Paper Revisions

We again thank the reviewers for their feedback and want to clarify that in addition to supplementary materials that are now available, we have also posted a revised draft that reflects the following changes (reviewers with relevant addressed feedback in parens):

  • Minor edits for clarity
  • Cleaned sources in bibliography for more consistency
  • Spacing adjustments (Reviewer 6WYD)
  • Emphasis of variance result of frontier replication in main body (Footnote 8) (Reviewer 6WYD)
  • Noting of hand-picking and random sampling of illustrative examples (Section 5.2) (Reviewer UbEG)
  • Footnotes and pointers to new appendix sections (see below)
  • Appendices
    • Movement of illustrative examples from C.1 to E.5
    • Renaming E.6 to "Additional Qualitative Analysis"
    • Changing text and table sizing in Figure 6 (Gemini judge prompt) and wide qualitative example tables
    • New Appendix E.3 detailing attack diversity experiments (Reviewer rqpn)
    • New Appendix E.4 detailing counterfactual instruction experiments (Reviewers 6WYD and UbEG)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.