acceptodds
Under review as a conference paper at ICLR 2027

The Hidden Fragility of AI Safety: How Guardrails Can Weaken Cyber and Biological Resilience

Abstract

Safety of powerful AI systems is increasingly pursued through capability restrictions by way of model guardrails. These guardrails try to restrict models from providing assistance with offensive cyber operations, viral engineering, or other capabilities which the model providers judge to pose elevated risk of misuse. However, such framing evaluates safety at the level of an individual model and user interaction, without taking into account how restrictions alter the capabilities and incentives of other actors. By examining these systems through the lens of resilience, we argue that model guardrails should be evaluated not by how much capability they remove from an individual model, but by how they affect the resilience of the broader system in which that model operates. We present counterarguments for approaches to safety based on model guardrails, highlighting their trade-offs and their potential to weaken resilience of systems they're supposed to protect.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI safety, model guardrails, cybersecurity, biosecurity, dual-use, capability restrictions

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6tg4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper discusses the fact that most LLM companies use guardrails to prevent security issues, however, such intervention may influence normal use for ordinary people.

Critical strengths
  1. The paper shifts the AI safety discourse from static, single-turn model alignment to dynamic, system-level resilience over time.
  2. The arguments are well-supported by recent, concrete examples and empirical benchmarks. These include the July 2026 Hugging Face intrusion incident—where commercial frontier guardrails obstructed active forensic analysis and forced defenders to fall back on a less capable open-weight model。
Critical weaknesses

While the paper convincingly diagnoses the flaws of current guardrails, it offers very little in terms of concrete, actionable solutions. Beyond broad calls in the conclusion to strengthen underlying physical/software infrastructure and include legitimate research preservation as an evaluation objective, the paper does not specify how model providers should redesign alignment training, refusal classifiers, or access-control mechanisms to resolve the defender-attacker asymmetry.

Questions for authors

See weaknesses

2 comments

Thank you for the review. We agree that the paper does not propose a concrete redesign of guardrails, but this is not its intended contribution. Our goal is to challenge the prevailing assumption that preventing harmful model responses necessarily improves overall safety, particularly when restrictions also obstruct legitimate defensive work.

We believe identifying this overlooked tradeoff and questioning whether evidence supports the assumed safety benefit of guardrails is itself a meaningful contribution.

Most importantly the paper argues that guardrails may themselves exacerbate the defender–attacker asymmetry by restricting legitimate defenders while attackers retain access to unrestricted capabilities.

Thank you for your clarification, I decide to maintain my score

Reviewer_rVnLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper is mainly focusing on evaluating AI guardrails , especially their impact on the overall cyber-biological resilience of a system, as well as the potential for restrictions to impede legitimate research and defensive capabilities.

Critical strengths

This work provides two goals of safety evaluation: preventing harmful responses and maintaining defensive capabilities.

Authors also link defensive refusals to researchers' access to information, distributed expertise, and preparation prior to an incident, while recognizing the importance of preventing misuse and irreparable biological harm.

Critical weaknesses
  1. The primary mechanism of resilience is not evidence-based: While there is evidence that guardrails can prevent some legitimate actions, this does not demonstrate that such restrictions will lead to an accumulation of systemic fragility. Instead, the authors provide examples of trajectories rather than establishing a link between refusals and delays in remediation, decreased preparedness, or increased loss from incidents.
  2. The offense-defense comparison is incomplete: Figure 2 is based upon an assumption of greater defensive improvement with no guardrails while maintaining offensive capability constant.Therefore, it illustrates a potential defensive cost rather than establishing the conditions under which restrictions will result in overall safety increaseWhat is the connection between guardrail induced refusals and long-term decreases in defensive preparedness or increases in loss of incidents? Under what circumstances will the defensive benefits of reduced restrictions outweigh the increased offensive capabilities? How would you assess broad access vs. trusted access vs. controlled testing vs. more context-aware guardrails? What evidence exists to support the claim of an asymmetric relationship between AI assistance for biological attackers and biological defenders? What are some measurable outcomes that would allow for testing of the proposed resilience framework?.
  3. The authors do not adequately address when these alternative methods are ineffective.
  4. The biological conclusions are unsupported by with evidence: Although bioterrorism has historically been rare and there are many practical barriers to biological attacks, these facts do not demonstrate that advanced AI will provide less marginal assistance to attackers than to defenders.
  5. The proposed evaluation framework is not operationalized: The authors group refusals, classifiers, model fallbacks, and access restrictions together but do not identify measurable resilience outcomes or a decision-making rule for evaluating different guardrail policies.
Questions for authors

1)What is the connection between guardrail induced refusals and long-term decreases in defensive preparedness or increases in loss of incidents? 2)Under what circumstances will the defensive benefits of reduced restrictions outweigh the increased offensive capabilities? 3)How would you assess broad access vs. trusted access vs. controlled testing vs. more context-aware guardrails? 4) What evidence exists to support the claim of an asymmetric relationship between AI assistance for biological attackers and biological defenders?

2 comments

Thank you for the review.

Our central idea is to challenge the prevailing assumption that preventing harmful model responses necessarily translates into improved overall safety. We acknowledge the benefits of misuse prevention, but we also present empirical evidence that guardrails obstruct legitimate defensive work. Where, then, is the corresponding empirical evidence demonstrating that the benefits of these restrictions outweigh their costs to defensive capabilities? Figure 2 illustrates a potential tradeoff rather than an established causal relationship.

We agree that determining the net effects would require longitudinal, counterfactual evaluations across different access policies, institutions, and adversarial environments.

As for the questions not answered by above:

  1. The Hugging Face and OpenCode (later, not cited in the paper) incidents provide direct evidence of this mechanism on shorter timescales. In both cases, guardrails obstructed legitimate incident response, forcing defenders to seek alternative models and waste time undefended. The cumulative consequences of such restrictions are the open question our paper raises.

  2. Reduced restrictions may improve overall safety when attackers already have access to comparable capabilities through alternative models or jailbreaks, while legitimate defenders remain constrained. Conversely, restrictions may be beneficial when they meaningfully limit otherwise unavailable offensive capabilities, but this scenario is unknowable. Defenders do not posses information on the capabilities of the attackers. From perspective of resilience and from the defenders point of view, there is much more confidence in the defensive preparedness if it had been hardened using models with unrestricted capabilities.

  3. We discuss the tradeoffs between different access policies throughout the paper, particularly the limitations of trusted access for independent researchers and incident responders. We agree that this could be strengthened with additional evidence and statistics on access barriers.

  4. Regarding the biological offense–defense asymmetry, the absence of evidence for this asymmetry does not establish the opposite, nor does it demonstrate that guardrails are necessary or beneficial. Uncertainty should not be treated as confirmation of the prevailing approach, and we thought it was worthwhile to challenge the prevailing position. The only thing that is known from empirical evidence is that guardrails, in some cases, can prevent more legitimate work than malicious, as was shown by BioSecBench-Refusal (Wintermute et al., 2026).

Thank you for addressing some of my concerns but I will maintain my score due to some questions were unanswered.

Reviewer_b8ccOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper argues that model guardrails should be evaluated by their effects on system-level resilience, rather than solely by their ability to prevent harmful individual interactions. It identifies several potential mechanisms through which guardrails may weaken resilience. The paper supports its arguments with examples and findings from cybersecurity and biosecurity, while acknowledging the immediate misuse-prevention benefits of guardrails. Its central contribution is a conceptual reframing of AI safety as a trade-off between misuse prevention and long-term defensive preparedness.

Critical strengths
  • The paper moves beyond single-interaction refusal rates to consider the broader consequences of guardrails for defensive ecosystems, legitimate research, and preparedness against emerging threats.
  • Connecting cybersecurity and biosecurity through the concept of system resilience offers a useful framework for examining common challenges in dual-use AI governance.
Critical weaknesses
  • The proposed mechanisms and figures are explicitly illustrative rather than empirical. The paper does not establish whether guardrails improve or weaken overall resilience under realistic conditions, nor does it quantify the relevant trade-offs. The central claims are primarily supported by illustrative cases, cited studies, and qualitative reasoning. The paper does not provide a systematic empirical evaluation of the net effects of guardrails on system-level resilience.
  • The discussion of physical-world constraints and the relative benefits of unrestricted access may underestimate how AI capabilities can lower barriers to harmful activities. The paper needs a more balanced analysis of when guardrails may produce net benefits or net costs.
  • The causal link between restricted model access and reduced long-term defensive preparedness remains largely hypothetical. Other factors influencing resilience are not adequately isolated.
Questions for authors
  • How would you operationalize and measure system-level resilience, and what metrics would allow a meaningful comparison between systems with different guardrail policies?
  • Can you provide empirical evidence, a controlled study, or a systematic case analysis that establishes a causal relationship between guardrails and reduced defensive preparedness?
  • How does your framework account for the possibility that guardrails simultaneously improve defenders' access to safe assistance and restrict potentially harmful capabilities? Under what conditions would the proposed resilience costs outweigh the benefits of misuse prevention?
1 comment

Thank you for distinguishing the paper’s conceptual contribution from the empirical evidence needed to establish net effects.

Regarding the measurement of system resilience, we agree that the manuscript does not establish a comprehensive scientific framework for evaluating the systemic effects of guardrails, nor do we claim otherwise. The direct evidence we present concerns the obstruction of legitimate work, drawing on empirical studies and documented incidents already cited in the paper: Campbell et al. (2026) on cybersecurity defensive refusals and BioSecBench-Refusal (Wintermute et al., 2026), both discussed in Section 4.1, alongside the Hugging Face incident discussed in the Introduction and Section 4.2. We also explicitly acknowledge the security benefits of guardrails in Section 3.

We would also like to point out the asymmetry in the burden of evidence. Evidence that guardrails prevent certain forms of misuse does not, by itself, establish that they improve overall safety. Once their obstruction of legitimate defensive work has been empirically documented, why should the burden fall exclusively on those identifying these costs to demonstrate their full downstream consequences? Those introducing guardrails should equally be expected to actually providing scientific evidence of their utility in the stated goal. One of the main assumptions that we tried to challenge is the that the net safety benefit of guardrails follows directly from misuse prevention.

To be fair to both sides, establishing quality scientific evidence would require substantially more than model-user interaction benchmarks. It would require counterfactual evaluations of defensive capabilities, adversarial adaptation, and downstream security outcomes, potentially spanning multiple years, organizations, and model generations. A narrowly scoped experiment might demonstrate one particular effect, but it would not resolve the systemic question which is the main idea of our paper. We believe it would be misleading to present such an experiment as sufficient evidence for a conclusion of this scope.

This is precisely where we see the paper's conceptual contribution. Scientific progress does not consist exclusively of introducing new metrics or experiments, it also involves identifying overlooked mechanisms, challenging prevailing assumptions, and questioning whether existing evidence actually supports the conclusions drawn from it. Our contribution is to highlight a neglected dimension of AI safety evaluation, the potential erosion of the defensive capabilities on which long-term security depends, which is, in part, already empirically validated through the obstruction of legitimate work.

We would welcome feedback on how realistic experiments could be designed to evaluate these long-term effects on systemic resilience, or even narrower experiments that would make this paper a more meaningful contribution to the community.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.