acceptodds
Under review as a conference paper at ICLR 2027

Structurally Close, Security-Wise Far: Contradictory Policies in Neural Networks

Abstract

How close can two neural subnetworks be if they implement essentially the same benign functionality but contradictory security policies? We formalize this question through structural security distance (SSD), the minimum Hamming distance between masks of a common parent network that approximate a prescribed safe/unsafe policy pair. We also distinguish unrestricted SSD from distance under mask-size constraints and from the distance between a specified safe mask and the nearest unsafe realization, since these quantities need not agree. We establish four main results. First, structural proximity depends on which realizations are compared. We construct a perturbation-stable ReLU family in which the minimum-size safe and unsafe masks are \(2q\) edits apart, the minimum safe mask is \(q\) edits from the nearest unsafe realization, and the unrestricted SSD is a single mask-coordinate change. Thus, a globally close safe/unsafe pair does not imply that a particular deployed safe model is close to an unsafe realization. Second, low-dimensional safety localization does not imply a small number of structural edits: a one-dimensional safety direction can require \(\Theta(d)\) binary neuron-gate changes, and under additional assumptions on benign representations, every discrete one-gate-at-a-time transition can incur a constant benign-error barrier even though continuous directional ablation preserves benign behavior. Third, estimating normalized SSD remains NP-hard in the worst case even at fixed nonzero approximation tolerance, with guaranteed feasible safe and unsafe masks, known minimum sizes, bounded sparse inputs, and robustness to parameter perturbations. Finally, we show that any transition between opposite policies must accumulate sufficient functional change, yielding certificates for the safety of a deployed mask within a bounded number of edits.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
neural network security, lottery tickets, overparameterization, sparse subnetworks, security alignment, Mixture-of-Experts, routing security

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_TUngOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

Attempting to theoretically analyse the impacts of structural stability on safety.

Critical strengths

Theoretically, this work could be interesting. The authors are exploring the foundational mechanisms of how networks work, and are attempting to construct a theoretical framework around interesting concepts. The work contains some mathematical sophistry.

Critical weaknesses

When assessing a paper, I look at two primary avenues - how interesting is the core idea (and how well is it explored), and how well the work is presented. Unfortunately, I must say that to me, any ability to appreciate the work is hamstrung by the way the authors presentation. The work is littered with undefined terms and other writing issues (see below), and I must ask who this work is actually written for? Because I must admit that as someone from the AML community, I find this work almost impenetrable - not due to the mathematical detail, but rather the comprehensive lack of detail and descriptions in the authors presentation. This reads as a paper that was written either for a very specific audience or specifically for the authors themselves, with no regard to how it would be interpreted by the broader community. Which is a problem, because an idea should be broadly accessible.

I will also state that, and this may speak to my biases, but I find myself unconvinced by completely theoretical works at this point in time - especially when AI tools were used to "refine the conceptual framework...clarify limitations, and edit the manuscript". Without some level of experimentally testable hypothesis, I cannot be certain that the LLM that was involved in generating these theories was actually correct. And I have no way to calibrate the impact of the theoretical assumptions made by the authors.

It could be argued that peer review is the exact place for testing such things, and that reviewers should be doing this. However, this is offloading responsibility onto reviewers. I do not think that reviewers should be just skimming over papers and treating all results as true, but there is an element of reciprocity that is required within the peer reviewed process, and unfortunately, it's difficult to establish this for theory only papers.

With that said, my judgement is based strictly upon the content invested, rather than my concerns regarding reciprocity for theory-papers. However, I would suggest to the authors that there's value in attempting to experimentally elucidate theoretical contributions, at least on a toy-model level.

Writing issues

  • Abstract is....a lot. 9 sentences, 21 lines, including 1 sentence that runs 6 lines long. This is not aligned with the general accepted practice and style of papers at venues like ICLR. This comment is not presented to gatekeep, but rather to indicate that I view the deviation from standards of practice to be indicative of a style of writing that values density and verbosity over clarity.

  • Undefined terms - "same benign functionality but contradictory security policies" , the security policies aren't defined.

    • "strong variants"
    • "Multi-task supermasks and universal tickets"
    • "robust tickets, Trojan tickets, and unsafe/safety tickets"
    • "security-domain representations....the safety direction"
    • "security decisions"
    • "benign error"
  • Even the contributions are doing so much heavy lifting - they're explaining content, without defining parameters, without any of the associated context. Who is this meant for?

  • "Structural security distance" is treated as a proper noun with the introduction of the acronym, so it should have all leading letters capitalized.

  • Citation style issues. See for example Li & Kim's Superficial Safety Alignment Hypothesis .. (Li & Kim, 2026) would usually be presented as Li & Kim's (2026).

  • |. | isn't defined for the L_0 norm in Eqn 3, or at least it's only implicitly defined as the "number of active mask coordinates".

  • Parameters are often undefined - see for example the ranges in Theorem 3. There's no justification as to why these ranges exist?

  • Theoretical results are presented without the context to explain them. Theorems 6 & 7 and Prop 5 are given a 5 line framing (lines 281-285), but the framing is strictly focused on the "what", not the "why". There's no context here for anyone to understand why holding these theorems in their heads would be at all useful. The same goes for the entirety of Section 6.

Questions for authors
  1. Why is the hamming distance an appropriate measure for security distance?

  2. What parts of this work would be amenable to experimental validation?

1 comment

Thank you for the detailed feedback. We recognize that the original manuscript's presentation makes the contribution unnecessarily difficult to follow. We have rewritten the abstract and introduction, reduced specialized terminology, and added a worked example and experimental results to make the theoretical findings more accessible. We also moved some less central results to the appendix.

  1. Hamming distance. Hamming distance counts the number of binary structural decisions required to move between two subnetworks of the same parent. This is directly relevant for parameter pruning or disabling predefined components, such as neurons, attention heads, or adapters. It is not intended as a universal measure of security risk. An important finding of our work is precisely that proximity between feasible policies, distance from a deployed model, and distance to an arbitrary security violation are different quantities.

  2. Experimental validation. We have conducted an exploratory experiment on Qwen2.5-1.5B-Instruct comparing rank-one activation projection with binary gating of actual MLP intermediate neurons. Directional projection reduced detected refusals from 12/12 to 3/12 harmful prompts, whereas disabling 128 selected MLP neurons left 11/12 refusals. This demonstrates different behavioral effects under the two intervention classes. Still, it does not establish the minimum number of required edits or prove preservation of benign functionality.

  3. Presentation and interpretation. The revised manuscript explains the main theoretical results before introducing their formal statements, provides a concrete illustration of the structural-distance separation, and adds a computable certification example. We have also clarified the assumptions and limitations of the results, particularly the distinction between prescribed unsafe policies and arbitrary security violations.

Reviewer_An2mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the following problem: How close can two neural subnetworks be if they implement essentially the same benign functionality but contradictory security policies? This paper formalizes this question through structural security distance (SSD), the minimum Hamming distance between masks of a common parent network that approximate a prescribed safe/unsafe policy pair.

This paper then gives a series of theorems and lemmas which bound the SSD etc.

Critical strengths

The formulation of this paper or the question asked in this paper is interesting. It is mostly a theoretical paper, but with a neat problem.

Critical weaknesses
  1. The application background of the studied problem needs to be better explained.

For example, if this problem is solved, why do we care about this problem? How should we apply the results in practice?

  1. There are many theorems and lemmas, however, how important these theorems and lemmas are not clear.

    What usages do these theorems and lemmas have? What are their applications?

  2. There are no numerical results verifying how tight the bounds (upper or lower bounds) in these theorems and lemma are

  3. There are no numerical results showing the application of these theoretical results.

  4. Some notations may not be carefully defined, for example, math cal{R} is not defined when first used?

Also, some calligraphic A is written as a regular A.

Questions for authors
  1. Could the authors better explain the application background of the research problem?

  2. Could the authors draw a diagram or give a detailed example of the results in the lemmas or theorems.

  3. Could the authors give some numerical results?

2 comments

Thank you for highlighting the relevance of the research question. We have revised the introduction to explain the security setting and the practical meaning of the different distance definitions.

  1. Motivation. SSD measures how many binary component changes separate realizations of prescribed policies that agree on benign inputs but differ on security-sensitive inputs. This is relevant when comparing pruned subnetworks or analyzing the structural accessibility of unsafe behavior. SSD is not itself a general robustness certificate.

  2. Worked example. We added a numerical illustration and diagram for Theorem 3. With q=4q=4, minimum-size safe and unsafe masks differ by eight edits; the fixed minimum safe mask is four edits from the nearest unsafe realization, while unrestricted SSD equals one. The difference arises because a nonminimal safe mask can contain dormant circuitry that becomes active through a single edit.

  3. Experimental results. We conducted an exploratory experiment using Qwen2.5-1.5B-Instruct. On 12 examined harmful prompts, automatically detected refusals decreased from 12/12 to 3/12 after directional projection, while disabling 128 selected MLP intermediate neurons left 11/12 refusals. The MLP intervention also produced less benign-output distortion (JSD 0.037 vs. 0.099 for projection). These results illustrate differences between intervention classes, but do not establish successful harmful compliance, preserved benign functionality, or minimum SSD.

We have also corrected notation inconsistencies and clarified the purpose and assumptions of the main theoretical results.

Considering that the numerical results are still limited, I will maintain my current score.

Reviewer_s1bNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines structural security distance (SSD) as the minimum Hamming distance between masks of a common parent network that approximate prescribed safe and unsafe policies. It distinguishes unrestricted SSD, distance under mask-size budgets, and distance from a specified safe mask. The main results include an exact, perturbation-stable separation among these quantities; a separation between one-dimensional activation interventions and binary neuron-gate edits; worst-case additive approximation hardness; and a sufficient condition for preserving a deployed mask's safe sign under bounded edits.

Critical strengths
  1. The paper distinguishes global proximity from proximity to a deployed mask. The exact construction shows why a close safe/unsafe pair can coexist with a much larger distance from the particular safe model being audited.

  2. The main constructions address several easy objections. The size-budget separation survives parameter perturbations, and the hardness result retains guaranteed feasibility, known minimum mask sizes, sparse bounded inputs, and a fixed perturbation radius.

Critical weaknesses
  1. SSD measures distance to a prescribed unsafe target, which can differ substantially from distance to any security violation. This limits how directly a large SSD can support a deployment-safety conclusion.

  2. Several results follow from standard geometric or combinatorial arguments, while the stronger separations rely on specialized constructions and edit classes. The overall methodological advance needs a sharper justification.

  3. The deployed-mask certificate requires uniform bounds over both inputs and intermediate masks, but the paper does not show how to obtain useful bounds for a concrete nontrivial model family.

Major Concerns

  1. Distance to a specified unsafe policy is not the same as distance to a security failure.

The unsafe feasible set in Equation (9) requires approximation to one prescribed function fu over the entire domain X. The fixed-mask quantity in Equation (13) retains this restriction. A model may violate the safety predicate on part of R while remaining far from fu, or may implement a different unsafe policy with the same benign functionality. Such a model need not belong to the unsafe feasible set being optimized over.

This is a legitimate definition for studying realization geometry, and the paper states it clearly. The issue is its security interpretation. A lower bound on Dfixed does not generally establish that the deployed mask is far from any unsafe behavior. Corollary 10 addresses a broader safety property through a separate uniform-influence argument, but that should not be read as a consequence of large target-conditioned SSD.

  1. The theoretical contribution is uneven, and its relationship to existing distance problems needs more precision.

Theorem 2 follows from a Hamming triangle inequality once every near-minimum feasible mask is assumed to contain a minimum-size feasible submask. Theorem 9 is a telescoping output-distance bound, and Corollary 10 applies uniform toggle bounds and Lipschitzness. These are useful observations, but they are not substantial new mechanisms by themselves. The authors acknowledge the elementary arguments; the presentation should give correspondingly more weight to the exact separations and the hardness construction.

The strongest construction in Theorem 3 is informative, especially its fixed perturbation radius. However, the general size-budget theorem depends on a strong inclusion property that need not hold when different representations or cancellation mechanisms are available. The example itself deliberately shows where this property fails. It would help to identify a meaningful class of parents or specifications for which the property follows from checkable conditions, rather than having to assume a statement about all feasible masks.

The paper also cites prior work on minimum Hamming distance between propositional models. Please distinguish more explicitly which part of Theorem 8 is specific to neural-policy approximation and which part is a robust encoding of a constrained Boolean distance problem. I regard guaranteed feasibility and the normalized gap as useful refinements, but the security terminology alone does not make the underlying optimization problem new.

  1. The direction-versus-neuron result is compelling within its edit model, but its broader implications remain limited.

Theorems 6 and 7 compare a dense representation-space projection with binary coordinate gates in a fixed readout family. A distributed one-dimensional direction can require many coordinate deletions, and the paper quantifies this cleanly through coordinate concentration. However, this is also a comparison between different intervention classes. Continuous gate scaling follows the directional output path in this model, as the paper explicitly notes.

The benign-error barrier additionally requires quantitative coverage of every direction orthogonal to v. The observation that fewer than d-1 independent projected benign vectors cannot give a positive coverage constant is important. Beyond rank, the magnitude of that constant determines whether the bound is informative; near-degenerate coverage can make the barrier negligible. Security representations must also remain close to v throughout R.

These assumptions are sufficient, but the paper does not establish when they describe safety mechanisms of practical interest. I am not asking for a full LLM experiment in a theory paper. A substantive model class where the constants can be bounded, or a theorem for weaker distributional benign constraints, would make the connection more convincing. At minimum, the discussion should distinguish the basis-dependent coordinate-edit statement from a general claim about the structural localization of safety.

  1. The deployed-mask certificate is not yet an operational auditing method.

Corollary 10 requires each ce to bound a toggle's effect uniformly over all security inputs and every intermediate mask within the relevant Hamming neighborhood. This is much stronger than evaluating the influence of a coordinate at the deployed mask. The paper correctly warns that endpoint gradients do not supply the needed bounds.

The remaining question is how an auditor obtains them. Enumerating the neighborhood can be combinatorial, and coarse bounds may be too large to certify even one edit. The manuscript does not give a concrete nontrivial family where these constants can be computed efficiently and the resulting certified radius is useful.

A constructive instantiation would substantially strengthen this part of the contribution. For example, derive computable bounds for a specified architecture and bounded input domain, then characterize when their top-k sum remains below the security margin. Without that step, the corollary is a sound sufficient condition whose verification burden remains unresolved.

Questions for authors
  1. How does target-conditioned SSD relate to the minimum edit distance to any safety violation under a benign-utility constraint? Can a deployed mask have a large Dfixed but a one-edit violation that does not approximate the prescribed fu?

  2. Which checkable properties of a parent network guarantee the minimum-submask inclusion assumption in Theorem 2, and when are the coverage and concentration constants in Theorem 7 nonvacuous?

  3. Can Corollary 10 be instantiated with efficiently computable uniform toggle bounds and a useful certified radius for a concrete model family?

2 comments

Thank you for the detailed comments. We have revised the manuscript to distinguish the structural distance between prescribed policies from the distance relevant to deployment security.

  1. Target-conditioned SSD versus violations. We now explicitly define the minimum edit distance to any security violation under a benign-change budget. A simple construction illustrates the difference: realizing the prescribed unsafe policy requires nn edits, while a single edit can violate the security predicate without changing benign functionality. As such, large target-conditioned SSD does not imply deployment robustness. Corollary 10 provides a separate sufficient guarantee against any violation within a certified edit radius.

  2. Theoretical contribution and assumptions. Theorems 2 and 9 provide supporting bounds, while the main contributions are the exact perturbation-stable distance separation, the directional-versus-binary-edit distinction, and the fixed-tolerance hardness result. The minimum-submask inclusion assumption is not universal. We now provide a checkable sufficient condition using witness inputs that force specific gates to remain active. We also distinguish the combinatorial hardness inherited from exact-cover problems from the neural-policy encoding and its robustness properties.

  3. Representation assumptions and certification. The directional results apply to a fixed representation and edit class. For finite benign data, the coverage constant can be bounded through the smallest singular value of the projected representation matrix; near-degenerate coverage produces a weak bound. We additionally instantiate Corollary 10 for a bounded-input gated ReLU readout. Since its uniform toggle bounds are efficiently computable, then a six-gate example certifies safety against three edits, while four edits can violate the predicate. We do not claim that this provides tight certificates for general transformers.

Thank you for your reply. It addresses a few concerns, but some of your claims may not be sufficiently supported:

  1. I agree with your new example, and that is exactly my concern. However, the new "distance to any violation" looks like the quantity that matters for security, but apart from Corollary 10 I don't see results about it.

  2. The witness-input condition sounds useful, but I can't judge it from one sentence. Does it hold for any family other than the one you construct?

  3. The six-gate example is too small to answer my question. With six gates there are only 64 masks to check, so it doesn't show that the certificate helps where enumeration is out of reach. What I'd like to know is whether the toggle bounds stay useful as the width grows.

I think the paper would be stronger if it were built around the any-violation distance. Based on these concerns, I will keep my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.