acceptodds
Under review as a conference paper at ICLR 2027

KeyedRoPE: Structural Capability Control through Key-Conditioned Attention Geometry

Abstract

Can access to a language model's learned capabilities be made contingent on its internal attention geometry rather than an external textual credential? We investigate this question through KeyedRoPE, a structural capability-control mechanism that conditions rotary attention geometry on a compact nontextual key. KeyedRoPE transforms token positions and reconstructs RoPE phases, making useful model execution dependent on an authorized attention geometry. Across four language models from 410M to 8B parameters and four downstream tasks, protected models largely retain authorized utility while performance substantially degrades under standard RoPE and held-out incorrect keys. Under adaptive white-box evaluation, gradient-based geometry search yields partial capability recovery, while fine-tuning recovery increases with task-specific supervision and is slower than Password-Locked Models at limited and intermediate data budgets. In our Qwen3-4B Mobile Action evaluation, full-data adaptation recovers 96.2% of the pre-attack authorized utility and 92.13% of the original task-adapted utility under standard RoPE. These results do not establish that full-data retraining is necessary for recovery. Instead, they show that KeyedRoPE transforms direct checkpoint reuse into an explicit recovery and re-adaptation problem whose difficulty can be measured under specified attacker budgets.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Active Authorization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_BNoaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes controlling model capabilities through key-conditioned attention geometry rather than textual passwords. A 12-dimensional continuous key specifies control-point correspondences used to construct positional transformations and phase remapping. Protective training preserves task performance under the authorized key while suppressing performance under incorrect keys or standard RoPE. The evaluation covers authorized utility, incorrect-key behavior, standard-RoPE bypass, gradient-based key recovery, and fine-tuning-based recovery across several models and tasks.

Critical strengths

Moving capability control outside the input-token space is an interesting direction. Conditioning the positional computation provides a distinct mechanism from textual triggers and creates a concrete setting for studying structural access control.

The security evaluation goes beyond a small number of randomly selected incorrect keys. The paper includes gradient-based recovery and fine-tuning recovery at different data budgets, while Appendix A.4 additionally evaluates 120 perturbed keys. The authors also appropriately restrict their claims to the tested attacks and budgets rather than asserting cryptographic unbreakability (pp. 7–9 and 14–15).

The results show a substantial empirical separation between authorized and unauthorized task performance. For example, the original, authorized, and standard-RoPE MMLU scores for Qwen3-4B are 66.86, 64.21, and 23.55, respectively (p. 6, Table 2). This establishes that the structural condition affects usable performance under the evaluated protocol.

Critical weaknesses

W1. The white-box analysis should target functional attention geometry, not only recovery of the original key.

Equations (5)–(7) suggest that, when ε is sufficiently small, the frequency-dependent normalization approximately cancels out, making the normalized phases nearly identical across frequencies. Since the subsequent nonlinear mapping is shared, the construction may effectively reduce to a shared nonlinear positional warp, rather than a more complex frequency-dependent transformation.

This does not establish equivalence to standard RoPE or imply a successful bypass. However, it suggests that recovering the original key may be unnecessary: a white-box attacker could instead optimize a simpler, functionally equivalent positional geometry.

The authors should verify this approximation using the actual frequency range and reference length (pp. 4, 13), and compare key recovery against direct optimization of a compact positional warp under matched data and compute budgets.

W2. Low task scores should be separated from failure to produce the expected answer format.

The incorrect-key training branch uses random input spans as targets. Several standard-RoPE MMLU scores fall below the 25% uniform-guessing reference for four-option questions, including 17.27 for Qwen3-0.6B and 14.10 for Llama-3.1-8B (Section 3.4; p. 6, Table 2).

These results are consistent with capability suppression, but they may also include invalid-option generation or failure to follow the answer format. Below-chance performance does not, by itself, establish that the underlying knowledge or reasoning is inaccessible.

The paper should specify whether MMLU is scored through free-form generation, constrained option decoding, or option likelihoods. Reporting invalid-output rates and adding an option-constrained or likelihood-based evaluation would distinguish suppression of task capability from suppression of the current response interface.

W3. The recovery-cost interpretation lacks the most relevant public-base-model reference.

With full task-data fine-tuning, the standard-RoPE variant recovers to 65.31, approaching the pre-attack authorized score of 67.89 (p. 8, Table 6). The evidence therefore supports delayed recovery under the tested budgets rather than irreversible prevention of recovery. The paper largely acknowledges this boundary.

The remaining issue is the reference point for the claimed additional cost. Comparisons with Password-Locked are concentrated on one model/task setting, while the original unprotected model primarily appears as a full-data reference. Since the underlying base model is publicly available, a practically relevant alternative is to adapt that base model directly to the task.

At matched data and computation budgets, how does recovering the protected model compare with simply adapting the public base model? Adding this curve at a few already evaluated data fractions would make the economic and practical meaning of the protection substantially clearer without requiring a broad expansion of the benchmark suite.

Questions for authors
  1. Over the implemented frequency and context ranges, how accurately do Equations (5)–(7) reduce to φ_j(p) = ω_j ψ_K(p)? Does directly optimizing a compact effective position function reduce recovery cost?
  2. How is MMLU scored, and what fraction of unauthorized outputs are invalid? How much of the authorized–unauthorized gap remains under constrained options or option-likelihood scoring?
  3. Under matched data and computation budgets, what is the task-adaptation curve from the public base model? Does this comparison change the interpretation of the protection’s additional recovery cost?
Reviewer_KyqWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes KeyedRoPE, a method that controls access to a language model's capabilities through key-conditioned attention geometry. It modifies token positions and RoPE phases using a nontextual key. During training, the model learns to produce correct outputs with the authorized key and alternative outputs with incorrect keys or standard RoPE.

The authors evaluate the method on four models and four tasks. The results show that authorized performance is largely preserved, while incorrect-key performance drops substantially. The paper also studies gradient-based key recovery, fine-tuning recovery, and component ablations. The recovery experiments show that capabilities can still be restored with sufficient task-specific data.

Critical strengths
  1. I like the idea of using attention geometry rather than a textual password to control model capabilities. The proposed design offers a different way to make model behavior depend on an authorization condition.

  2. The authors evaluate four models ranging from 410M to 8B parameters and four downstream tasks. The results show substantial performance gaps between authorized and unauthorized execution. Table 7 is also helpful in understanding the roles of the two geometric components.

  3. I appreciate that the authors do not rely only on incorrect-key performance. They also evaluate gradient-based key recovery and fine-tuning attacks. In particular, Table 6 shows that substantial capability can be recovered with sufficient training data, which helps clarify the limitations of the proposed protection.

Critical weaknesses
  1. Section 3.1 assumes a white-box attacker who knows everything except the authorized key. However, Table 3 and Appendix A.1.1 disclose the exact authorized key used in the experiments. I understand that a deployed model could use a different secret key, but I am not sure how the reported security results should be interpreted for the published configuration. The experiments also appear to rely on one authorized key, making it difficult to assess whether the protection generalizes across different keys.

  2. In Eq. (9), the model is explicitly trained to produce alternative outputs under incorrect keys and standard RoPE. I wonder how much of the performance degradation comes from the proposed geometry and how much comes from this training objective. Table 7 studies the geometric components, but it does not separate their effects from the unauthorized-branch training.

  3. KeyedRoPE combines positional transformation, local deformation, and nonlinear phase reconstruction. While Table 7 provides useful component ablations, I am not sure whether such a complex design is necessary. A simpler keyed positional transformation or RoPE modification, trained with the same protection objective, might achieve similar results. This comparison would help establish the technical contribution more clearly.

  4. Table 6 shows slower recovery than Password-Locked Models under the evaluated protocol. However, I could not find enough details to assess whether the two methods use fully comparable training and recovery settings. More importantly, the paper does not compare recovery from the protected checkpoint with fine-tuning the original public base model using the same data and compute budgets. Without this comparison, I find it difficult to judge how much additional effort an attacker actually needs after stealing the protected checkpoint.

  5. I could not find complete information about the Mobile Action dataset construction, protection-training hyperparameters, and key-recovery optimization settings. The white-box evaluation also focuses on a limited set of supervised recovery budgets. Since the paper aims to make capability recovery more difficult, I think reporting the attack's actual computational cost and sensitivity to optimization budgets would strengthen the evaluation.

Questions for authors

Q1. Could the authors clarify how the authorized key is kept secret in the stated white-box setting, given that the experimental key is disclosed? Could the authors also evaluate multiple independently sampled authorized keys to show whether the protection depends on the selected configuration?

Q2. Could the authors evaluate KeyedRoPE without the incorrect-key and standard-RoPE training losses? I would like to understand how much protection comes from the geometric transformation itself rather than the alternative-target training.

Q3. Could the authors compare KeyedRoPE with simpler keyed RoPE transformations, such as positional shifts or frequency scaling, under the same protection-training objective? This would help clarify why the proposed nonlinear design is needed.

Q4. Could the authors provide more details about the Password-Locked baseline and clarify whether the recovery settings are matched? Also, how does recovery from a protected checkpoint compare with fine-tuning the public base model using the same data and compute budgets?

Q5. Could the authors provide the missing dataset and training details, together with the optimizer settings, number of steps, and computational cost of key recovery? It would also be helpful to see whether larger attack budgets substantially change the recovery results.

Reviewer_QZ14OpenReview ↗
Rating
2
Confidence
1
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

KeyedRoPE introduces a structural capability-control mechanism that conditions a language model’s rotary attention geometry on a compact nontextual key. It combines key-conditioned positional transformations and nonlinear phase reconstruction with protection training to preserve authorized utility while suppressing performance under incorrect keys or standard RoPE. Experiments across four models and four tasks demonstrate this behavior, while adaptive attacks reveal partial recovery through key optimization and substantial recovery with sufficient fine-tuning data.

Critical strengths
  1. Distinctive authorization mechanism. The paper explores attention geometry as an internal condition for capability access, offering an alternative to textual passwords.
  2. Evaluation across models and tasks. Experiments cover four models ranging from 410M to 8B parameters and four tasks, providing evidence of authorized utility preservation and unauthorized performance degradation across different settings.
  3. Explicit investigation of recovery limitations. The paper evaluates both gradient-based key recovery and fine-tuning attacks, and acknowledges that sufficient task-specific supervision can substantially restore capabilities without the authorized key.
Critical weaknesses
  1. Limited baseline comparisons. The paper compares KeyedRoPE with Password-Locked Models on only one model and one task. Comparisons with Secure Your Model (Findings of NAACL 2024) and The Elicitation Game (ICML 2025) would strengthen the evaluation, using the same backbone and matched attack budgets to assess both authorized utility and unauthorized capability recovery.
  2. Insufficient ablations to establish the necessity of the complex geometric design. The paper should include simpler alternatives, such as frequency scaling or nonlinear positional reparameterization, and separately ablate the local deformation field and nonlinear phase-shaping terms. These variants should be evaluated under matched training budgets and the same held-out incorrect keys to isolate their contributions.
Questions for authors
  1. Could the authors compare KeyedRoPE with Secure Your Model and The Elicitation Game, using the same backbone and matched attack budgets, to evaluate both authorized utility and unauthorized capability recovery?

Comments on the paper

1 comment
Area_Chair_31fZOpenReview ↗

Final Discussion Reminder

Dear Reviewers,

The discussion period is nearing its end. If you have not already done so, please review the authors’ rebuttal and follow-up responses. If you have any remaining questions or concerns, please leave a comment for the authors.

Thank you again for your time and efforts!

Best regards,
Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.