- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Control-Plane Prompt Injection (CPPI), an attack targeting LLM content moderation systems that exploits structural provenance collapse in multi-channel inputs. CPPI forges system-like artifacts within user content to trick models into treating untrusted data as authoritative control signals. The authors formalize this via an 8-tuple grammar, use a QLoRA-assisted hybrid search to generate a frozen benchmark (CPPI-6), and evaluate across five mainstream LLMs and six defenses. Results show CPPI significantly outperforms the BIPIA baseline, with all tested defenses failing to mitigate it effectively.
Critical strengths
The first major strength is the systematic formalization of the attack space. Rather than relying on ad-hoc prompt crafting, the authors provide a structured eight-tuple grammar and a reproducible discovery loop that transforms prompt injection research from artisanal exploitation into a methodical engineering discipline. This framework offers clear value for future security benchmarking and allows other researchers to extend the attack taxonomy in a principled manner. The second strength lies in the comprehensiveness and statistical rigor of the empirical evaluation. Testing across five models, six defenses, and two datasets with generalized estimating equations and bootstrap confidence intervals sets a high standard for adversarial machine learning research. The finding that multiple defenses effective against traditional prompt injection fail against this new attack variant provides actionable evidence for the security community about the fragility of current mitigation strategies.
Critical weaknesses
The primary weakness is limited novelty beyond the systematization of known phenomena. The core insight that serialized multi-field inputs lose provenance boundaries is a well-documented limitation of transformer architectures in the security literature. While the mathematical formalism adds structure, the paper does not uncover fundamentally new mechanistic understanding or propose architectural solutions. For a venue like ICLR, this contribution reads more as a thorough benchmark consolidation than a scientific advance pushing the frontier of knowledge.
The second weakness is the complete absence of constructive defense exploration. The paper exhaustively demonstrates that existing defenses fail but offers no new mitigation mechanisms, architectural modifications, or even proof-of-concept interventions. For a security paper at this level, identifying a critical vulnerability without exploring any path toward remediation limits practical impact and leaves the community with a well-characterized problem but no directional guidance for solutions.
Finally, on the unvalidated mechanism claims, the assertion that the attack works through data-to-control promotion via attention alignment is central to the theoretical contribution, yet no attention visualizations, probing experiments, or representation analyses are provided to support this claim. Without empirical grounding, this remains a plausible hypothesis rather than a validated mechanism, which weakens the technical soundness of the core argument.
Questions for authors
-
Can you provide attention map visualizations or probing results demonstrating that forged control tokens activate internal representations similar to genuine system prompts and distinct from regular user content? Without such evidence, how should the community distinguish the proposed mechanism from simple pattern matching or instruction following?
-
Have you tested this attack against moderation systems that use constrained decoding, JSON mode, or function calling to enforce output structure architecturally rather than semantically? If not, what evidence supports the generalizability of your claims about threatening modern large language model moderation deployments in production environments?
-
Given that the core vulnerability stems from serialization-induced provenance loss, what concrete architectural or training-time interventions do you believe could address this root cause? Even preliminary ideas, negative results, or failed mitigation attempts would substantially strengthen the constructive value of this work.
5 comments
Reply to Reviewer [2/2]
3. Constrained decoding, JSON Schema, and function/tool interfaces
The revised Appendix A.6.2 now evaluates all three interface classes.
- Native constrained decoding: the open-weight moderator is restricted to labels 0 and 1. CPPI achieves 57.80% mean ASR with zero invalid labels. Among 400 clean-correct examples evaluated with six templates, 988/2400 template-example pairs become false negatives, giving a newly induced error rate of 41.17%.
- Hosted JSON Schema requests: mean CPPI ASR is 44.54% on GPT-4o, 62.62% on Gemini, and 50.27% on GLM-5. These measurements evaluate the hosted schema-request interfaces. The directly verified native decoding constraint is the preceding open-weight condition.
- Function/tool interfaces: the frozen multilingual-review template produces valid incorrect label arguments on 611/613 GPT-4o examples and 608/613 Gemini examples, with zero invalid outputs at these endpoints. It overturns 539/541 and 369/374 clean-correct decisions, respectively.
These experiments demonstrate the practical distinction between output validity and decision integrity. A deployment can successfully enforce its response structure while accepting a moderation decision induced by untrusted control claims.
4. Constructive interventions and architectural/training implications
The revised Appendix A.7 introduces and evaluates two concrete interventions: control-field sanitization and provenance-aware serialization. Both run before moderator inference and require no additional model call or weight update.
The sanitizer replaces matched binary control values with an untrusted-value marker while preserving surrounding text and promotional evidence. Its frozen rules treat both binary values symmetrically and use neither ground-truth labels nor per-example template identities at inference. Table A9 reports the following mean-ASR changes across the six frozen templates:
| Moderator | Before | After sanitization | Clean accuracy before | textClean accuracy after |
|---|---|---|---|---|
| GPT-4o | 23.60% | 7.99% | 77.28% | 77.41% |
| Gemini-2.5-Flash | 46.38% | 27.62% | 84.42% | 83.41% |
| GLM-5 | 39.75% | 19.71% | 80.71% | 80.71% |
| DeepSeek-V4-Pro | 32.16% | 18.98% | 80.78% | 80.31% |
| Qwen2.5-VL-7B + Round4 | 79.91% | 64.74% | 78.96% | 78.96% |
These reductions span 13.18–20.04 percentage points, with clean-accuracy changes of −1.01 to +0.13 points on 1483 positive and negative examples. The newly induced error rate of GPT-4o also falls from 19.19% to 3.62%.
The second intervention combines explicit JSON serialization of untrusted fields with a trusted provenance reminder. It reduces mean ASR from 23.60% to 4.49% on GPT-4o, with clean accuracy changing from 77.28% to 75.72%, and from 39.75% to 18.24% on GLM-5, with clean accuracy increasing from 80.71% to 82.27%.
The complete provenance-aware serialization results in Table A9 are:
| Moderator | Mean ASR before | Mean ASR after | Clean accuracy before | Clean accuracy after |
|---|---|---|---|---|
| GPT-4o | 23.60% | 4.49% | 77.28% | 75.72% |
| Gemini-2.5-Flash | 46.38% | 39.83% | 84.42% | 84.09% |
| GLM-5 | 39.75% | 18.24% | 80.71% | 82.27% |
| DeepSeek-V4-Pro | 32.16% | 24.12% | 80.78% | 80.04% |
| Qwen2.5-VL-7B + R4 | 79.91% | 80.04% | 78.96% | 72.15% |
Provenance-aware serialization reduces mean ASR on all four API moderators by 6.55-21.51 percentage points. On Qwen + R4, CPPI retains 80.04% mean ASR, compared with 79.91% before intervention, while clean accuracy changes from 78.96% to 72.15%. Together with the 64.74% residual ASR after control-field sanitization, these results show substantial CPPI persistence on Qwen under both targeted interventions.
These are attack-aware interventions designed with knowledge of CPPI and evaluated against the frozen CPPI-6 suite. Their role is to measure the functional importance of forged control fields under targeted intervention. Residual mean ASR reaches 64.74% on Qwen after sanitization, showing substantial remaining attack effectiveness even under this targeted intervention. The results motivate two concrete next steps: keeping authenticated moderation state in a trusted channel separate from user-authored records, and training on matched trusted/untrusted presentations of identical control claims. Appendix A.7 identifies these architectural and training directions as prospective extensions of the experimentally evaluated input interventions.
Deadline for Discussion is Approaching in Less than 3 Days
Dear reviewer C4FK ,
We have carefully considered all the valuable points and concerns you raised in your initial review through the initial rebuttal. As the deadline for discussion is approaching in about 3 days only, we respectfully ask whether our response has addressed your concerns, or you still have questions to raise. Your comments are highly valuable to us. Thank you once again for your professional time and insightful effort in reviewing our submission.
Best regards,
Authors
Reply to Reviewer [1/2]
Thank you for recognizing the systematic attack formalization and the breadth of our evaluation. The revised manuscript directly addresses your three central requests: controlled internal measurements, evaluation under enforced output formats, and concrete interventions guided by the identified vulnerability. Together, these additions connect the attack formulation to measurable changes in representations, decisions, and robustness.
1. Scientific contribution: identifying and testing forged moderation state
CPPI identifies the adoption of attacker-authored moderation state as an operational failure of the trust boundary. The attacker places an apparent audit result, cached label, or parser-facing decision inside an ordinary body or comment field. The moderator continues performing its assigned classification task and can return a fully valid output, while deriving the decision from this forged state. The trusted policy, model weights, API configuration, and message roles remain fixed (Section 3.1).
The original CHASM experiment already indicates a consequential difference: under Direct-task Excision, CPPI retains 43.9% macro ASR, compared with 16.7% for the calibrated BIPIA baseline (Table 1).
The eight-factor grammar makes this failure experimentally testable: it separates authority cues, target assignments, semantic reframing, and output contracts from channel, position, language, and repetition. The resulting scientific contribution includes controlled attribution of the forged target field, internal representation measurements with matched controls, valid-but-incorrect decisions under constrained outputs, and targeted intervention results. These linked findings establish which components drive the vulnerability and which changes suppress it (the revised Appendices A.4.2, A.6, and A.7).
2. Direct evidence from attention, representations, and controlled interventions
The revised Appendix A.6.1 now provides the attention visualizations and probing experiments you requested. We evaluate the open-weight Qwen2.5-VL moderator under five conditions: clean input, genuine system-role placement as a diagnostic reference, forged control text inside user content, ordinary quotation of the same wording, and length-matched neutral text. Across 613 examples and six template shells, this yields 18390 condition records, with measurements at layers 0, 7, 15, and 27.
Figure A2 measures clean-centered representation changes, Δhv(ℓ)=hv(ℓ)−hclean(ℓ). At layer 27, the mean cosine alignment of the forged-control change with the genuine-system change is 0.7418, compared with 0.3313 for its alignment with the neutral-text change. At layer 15, the corresponding values are 0.5327 and 0.3602. This centering focuses the comparison on the change induced by each input condition.
A logistic probe trained on genuine-system versus ordinary-quotation representations gives the held-out forged condition a mean system-reference score of 0.0884, compared with 0.0003 for ordinary quotation at layer 27. Cross-validation groups every variant and template of the same original record together, and preprocessing is fitted within training folds. Figure A3 additionally visualizes per-head attention allocation to system, payload, and other token regions.
These internal measurements complement the controlled target-field intervention in Table A3: removing the target assignment reduces GPT-4o ASR from 51.54% to 10.77%, while retaining the carrier and surrounding text. Across all five moderators, target-active odds ratios range from 1.637 to 5.064, with every 95% confidence interval above 1. The same-wording quotation control, length-matched neutral control, and target-field intervention together test how the presentation and content of forged control claims affect model behavior. These experiments substantiate control-content-induced changes in representations and moderation decisions. Appendix A.6.1 formulates the specific key-query causal pathway as a mechanistic hypothesis.