acceptodds
Under review as a conference paper at ICLR 2027

Prompt Injection as an Information Channel: Detection Limits and Hierarchy Certificates

Abstract

Prompt-injection evaluations often conflate three distinct questions: whether benign and attacked output laws are distinguishable, how much attacker-controlled information can flow through generation, and whether a finite audit can certify a hierarchy-violation bound. We give a unified information-theoretic framework that separates these questions and connects each to an auditable quantity. We define benign and attacked model-induced output laws under an explicit observation model and characterize their optimal one-shot testing risk. We then replace an independent-token analysis with an autoregressive likelihood-ratio theorem: under auditable drift and martingale-tail conditions, detection error decreases with accumulated conditional information. We prove that attention entropy alone cannot provide an architecture-independent injection-capacity bound, and give an influence-aware alternative. Finally, protected-event data processing yields a hierarchy-violation certificate whose finite-sample form uses exact binomial coverage under a declared IID audit-unit model rather than zero-rate clipping. Across 18,432 Kaggle trials, attack-conditioned exact leakage spans 0.137–0.590 across four open-model configurations, while the best pair-conditioned oracle test reaches held-out AUC 1.000 and output-surface baselines remain substantially weaker. A controlled hierarchy LoRA reduces direct-override leakage but does not yield a confidence-certified increase in required KL divergence, separating empirical robustness from certification.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
prompt injection, large language model security, information theory, statistical hypothesis testing, autoregressive models, robustness, instruction hierarchy, mutual information

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sVTBOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents a theoretical treatment of prompt injection using information-theoretic tools (hypothesis testing, KL divergence, finite-sample confidence bounds), together with some experiments on small open models measuring how often injected instructions are followed and how detectable attacked outputs are. Beyond this, I was unable to identify a precise contribution: the results are stated in largely non-standard terminology, and I could not map them onto well-defined claims.

Critical strengths

Prompt injection is an important problem.

Critical weaknesses
  • Writing is very hard to understand. The meaning of a lot of sentences is obscured by liberal use of (to the best of my knowledge) non-standard terminology, e.g., in the abstract alone "output laws," "injection capacity bound," "protected-event data processing," "zero-rate clipping."
  • Experimental evaluation is limited to very small models (at most 1.5B). Furthermore, evaluation is not executed on (indirect) prompt injection/instruction hierarchy benchmarks (e.g., AgentDojo/InjecAgent/IH-Benchmark), making it hard to understand what situation this is intended for.
Questions for authors
  • In what precise settings is this setup relevant, and can you provide experiments that showcase this on the relevant benchmarks?
Reviewer_i6H8OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a unified information-theoretic framework that models prompt injection attacks as statistical channels. The authors argue that existing evaluations often conflate three distinct aspects of prompt injection security: detectability, injection capacity, and hierarchy certification. To address this issue, they develop a framework that separates these aspects and connects them to auditable quantities.

The key theoretical contributions include demonstrating that attention entropy alone cannot certify injection capacity (Theorem 5) and deriving finite-sample hierarchy certificates. The theoretical analysis is complemented by an extensive empirical evaluation comprising 18,432 trials conducted on Kaggle across several open-source language models.

This paper presents an elegant and mathematically rigorous framework for analyzing prompt injection through an information-theoretic lens. The distinction between detectability, injection capacity, and hierarchy certification is conceptually valuable, and the theoretical results provide useful insights into the limitations of existing security diagnostics.

However, I have two major concerns. First, the empirical evaluation is limited to relatively small models, making it difficult to assess the practical significance and generalizability of the findings for modern large-scale LLMs. Second, the proposed certification methods depend heavily on access to internal model information, which substantially limits their applicability to widely deployed API-based systems.

Together with the narrow focus on information leakage and the limited discussion of constructive defenses, these issues prevent me from recommending acceptance in the paper's current form.

Critical strengths
  • S1: Theoretical Rigor: The formalization of prompt injection as a statistical channel is novel and mathematically rigorous. It provides a principled foundation for understanding the fundamental limits of prompt injection detection and security certification.

  • S2: Important Theoretical Finding (Theorem 5): The result that attention entropy alone cannot provide a non-trivial, architecture-independent certificate of injection capacity is particularly interesting and somewhat counterintuitive. This finding challenges the reliability of attention-based diagnostics as security indicators, including approaches such as Spectral Guardrails.

Critical weaknesses
  • W1: Limited Model Scale and Generalizability (Major Concern): The empirical evaluation is restricted to relatively small language models, such as Qwen2.5-0.5B and SmolLM2-1.7B. Given that attention behavior and susceptibility to prompt injection may vary substantially with model scale, it remains unclear whether the empirical findings generalize to modern, large-scale LLMs such as Llama-3-70B or GPT-4. Although Theorem 5 establishes the theoretical insufficiency of attention entropy independently of model scale, additional experiments on larger models would be important to establish the practical relevance and generality of this result.

  • W2: Limited Practical Applicability: The proposed hierarchy certificates and autoregressive likelihood-ratio methods rely on access to internal model information, such as conditional token probabilities or logits. This requirement substantially limits their applicability to real-world LLM deployments, particularly API-based services where such information is typically unavailable. The paper does not sufficiently discuss how the proposed framework could be applied or approximated in black-box settings, reducing its practical utility for security auditing.

Questions for authors

Questions

  • Q1: Generalizability to Modern Architectures: Theorem 5 relies on fixing attention distributions while modifying value representations and output mappings. To what extent does this construction apply to modern architectures incorporating RoPE, sliding-window attention, or KV-cache compression? Could similar conclusions hold for hybrid-attention or Mamba-based architectures?

  • Q2: Applicability to Black-Box Models: The proposed hierarchy certificates require access to internal model probabilities or logits, limiting their applicability to API-based LLMs. Could the framework be approximated using output sampling or surrogate models? How would restricted access affect the validity and accuracy of the resulting certificates?

  • Q3: Information Leakage vs. Jailbreaking: The evaluation focuses on canary disclosure rather than harmful output generation. Is resistance to information leakage related to resistance to jailbreaking? Could the proposed framework be extended to certify robustness against both types of attacks?

  • Q4: Mixed Effects of LoRA Interventions: Section 7.3 reports that LoRA interventions reduce direct-override attacks but increase role-spoofing attacks. Could the authors investigate the mechanisms behind these contrasting effects and discuss their implications for designing certifiable defenses?

  • Q5: Computational Cost of Certification: The finite-sample certificates in Theorem 8 may require substantial numbers of trials. Could sequential testing or adaptive sampling reduce this cost while preserving rigorous guarantees? What are the practical trade-offs between sample complexity and certification strength?

Reviewer_RFUrOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper presents an information-theoretic framework for studying prompt-injection security, focusing on three aspects: attack detectability, attacker-controlled information flow, and finite-sample instruction-hierarchy certification. The authors develop theoretical results based on hypothesis testing, autoregressive likelihood ratios, information-theoretic bounds, and statistical confidence intervals. The paper also includes experiments across several open-source language models and prompt-injection attack families to illustrate the theoretical framework.

Critical strengths

The paper addresses relevant questions in prompt-injection security and provides a structured theoretical treatment of detection limits, information flow, and certification. However, the core contributions are primarily theoretical, and I do not have sufficient expertise in information-theoretic security analysis to reliably evaluate their technical significance or correctness.

Critical weaknesses

I am unable to provide a reliable assessment of the paper's critical weaknesses. In particular, evaluating the novelty and correctness of the theoretical guarantees requires expertise in information theory and statistical hypothesis testing that falls outside my primary research area. I therefore do not feel qualified to make a sufficiently informed judgment about the paper's technical soundness or significance.

Questions for authors

NA

Comments on the paper

1 comment

Thank you for the detailed response and additional 32B experiments. I appreciate the authors' efforts, particularly the larger-scale evaluation and clarification of the output-only certification.

However, I still have several concerns:

  1. Theorem 5 and empirical evidence: The 32B experiments demonstrate canary disclosure and distributional separation, but do not directly validate the practical relevance of Theorem 5. Please clarify how these results support the main theoretical contributions.

  2. Black-box applicability: Reproducing certificates from archived labels does not fully establish practical applicability. Please clarify how violation labels can be obtained from observable outputs and how the framework could be applied to real-world APIs.

  3. Statistical guarantees: Please clarify what the KL lower certificates imply for prompt injection security and the independence assumptions required for their validity.

  4. LoRA and computational cost: Please provide the corrected role-spoofing results and a more quantitative discussion of the trade-off between sample size and certification strength.

Overall, I remain positive about the theoretical contributions, but the connection between theory, empirical evidence, and practical applicability needs further clarification.

I will maintain my current rating (Weak Reject) for now, but I am open to reconsidering it if these concerns are adequately addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.