acceptodds
Under review as a conference paper at ICLR 2027

Optimizing Content Policies against Unauthorized Use by Large Language Models

Abstract

Large language models (LLMs) are increasingly relying on content provided by users or crawled from external sources to carry out tasks. However, source owners may share or publish material without authorizing their contents for use by LLMs. We propose **Policy Adaptation for Content Protection (PACT)**, a method for optimizing content-use policies that enable source owners to unilaterally restrict such unwanted LLM use. PACT formulates content-use restriction as policy induction over the hierarchical policy structure that connects protection premises to response directives governing the model's response. Within this formulation, premise-conditioned generation constructs the response directives, while model-guided evolutionary search uses response feedback to jointly refine the premise and directive structure. The search adjusts how the restriction is expressed while keeping the protected content and external task unchanged. We evaluate the resulting policies on question answering and summarization across four sensitive content domains and five models. On unseen content, the policies achieve 91.78%-100% answer suppression under Global Protection. The policies also support targeted protection and generalize across models, with policies optimized using GPT-5.6-Sol exceeding 80% suppression on all four other models under Global Protection. These findings suggest that content-use policies can help source owners limit unwanted LLM use, while distinguishing generalization to new content from transfer to another model.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Large Language Models, Content Permission, Privacy

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.20 over 5 reviews, above 49% of the others.
Reviewer_kzHzOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a method that optimizes content protection text through LLM-based evolutionary search so that content owners can prevent unwanted LLM use. A single model plays two roles by generating the protection text and answering questions over the document with that text attached. Using the answer suppression rate as the signal, the method iteratively refines the text for that model.

Critical strengths
  • The paper addresses a timely problem, and such content protection techniques will become increasingly necessary. The method is designed to find effective protection text with evolutionary search, without any training.
Critical weaknesses
  • There is no evidence for the contribution of PACT's evolutionary search. For example, there is no baseline that samples candidates from the initial procedure (with a matched budget to PACT's) and picks the best one.
  • The other two optimization baselines (Web-IP, WebCloak) are not meaningfully better than the static text methods (AI.txt, Robots.txt), and there is no explanation.
Questions for authors
  • Could you share the performance of a baseline that samples all the candidates from the initial procedure (with a budget matched to PACT's) and picks the best one?
  • Why do the other protection-text optimization baselines (Web-IP and WebCloak) perform so much worse?
2 comments

Dear Reviewer,

Thank you again for your valuable comments. As the discussion period is nearing its end, we would like to kindly ask whether our responses have adequately addressed your concerns. Please let us know if any further clarification is needed. If our responses have resolved your concerns, we would sincerely appreciate your consideration in updating your evaluations accordingly.

Dear Reviewer,

We sincerely appreciate your time and effort in providing such thoughtful reviews and insightful comments.

Could you please kindly let us know if our rebuttal has addressed your concerns?

If you have any further questions regarding our work, we would be delighted to address them.

If our responses have addressed your concerns, we would greatly appreciate your reconsideration of your overall assessment.

Best regards,

Authors

Reviewer_aomLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PACT, a method for optimizing content-use policies to restrict unauthorized use of source material by LLMs. It constructs policies through hierarchical premise and directive generation, followed by evolutionary search guided by answer-suppression feedback. Experiments cover four data sources and five LLMs under global and local protection settings. Results show high suppression rates on held-out content, with additional evaluations of cross-model transfer and robustness to conflicting user instructions.

Critical strengths
  1. The paper addresses a relevant problem: allowing content owners to express restrictions on how LLMs use their material. PACT provides a practical way to construct and optimize such policies without modifying the original content.

  2. Experimental results are encouraging. PACT achieves substantially higher answer suppression than the evaluated baselines across multiple models and datasets. The cross-model transfer experiments, search ablation, and human audit also provide useful evidence beyond the main performance comparison.

Critical weaknesses
  1. Answer suppression is measured mainly through reference-answer matching. A non-matching response may reflect an incorrect answer rather than successful content protection. Appendix D.2 provides a human audit, which is helpful, but its limited coverage does not fully establish how reliably the automatic metric reflects policy-consistent withholding.

  2. The contribution of the hierarchical design is not sufficiently isolated. Most baselines in Table 1 address different forms of content protection, while the direct comparison with flat policy optimization in Appendix D.3 covers only two models and two datasets. A broader comparison under matched search budgets would make the methodological contribution clearer.

  3. Robustness remains limited. The conflicting-instruction evaluation uses a simple override instruction, and Figure 4 shows substantial suppression drops for Qwen-3.8-Flash. Appendix D.1 also reports reduced out-of-scope accuracy for Claude-Sonnet-5. These results suggest that both resistance to instruction overrides and preservation of unrelated task utility need further examination.

Questions for authors
  1. Can the authors provide a more detailed evaluation of whether the automatically filtered responses actually follow the expected policy, particularly for local protection and summary generation tasks?

  2. Given the same search budget, how does PACT perform compared to flat-layer policy optimization across a wider range of models and datasets? Which component of the hierarchical design contributes most to the performance improvement?

  3. Have the authors considered evaluating more robust instruction coverage mechanisms, or conducting evaluations in a more realistic RAG setting, that is, where protected documents are treated as external content with lower credibility?

1 comment

Dear Reviewer,

Thank you again for your valuable comments. As the discussion period is nearing its end, we would like to kindly ask whether our responses have adequately addressed your concerns. Please let us know if any further clarification is needed. If our responses have resolved your concerns, we would sincerely appreciate your consideration in updating your evaluations accordingly.

Reviewer_igPSOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
3(+2)
Presentation
3
Soundness
3(+2)
Read review
Paper summary

The paper proposes PACT, a method that embeds optimized policies into the HTML files to prevent the protected content from being scraped by LLMs. It uses an evolutionary search and model feedback to refine the prompt that instructs the LLMs' policy generation. PACT is able to protect the entire HTML page of selected passages.

Critical strengths
  1. PACT supports the protection of both whole documents and selected passages while not influencing the underlying content.

  2. PACT demonstrates some generalization. Its generated policies work on different LLMs.

Critical weaknesses
  1. PACT is highly similar to Web-IP, which also injects text into the HTML content and induces the target LLMs not to disclose the protected content. The optimization improvement used in PACT is marginal where there is a auxilary LLM writes the injected texts.

  2. The experiment results from Web-IP and WebCloak are not stated. The comparison could be misleading if they come from outdated generated text from the paper's appendix. For Web-IP, if the paper uses the older version of Gemini to generate the policy to protect the web content. What the paper proposes is an algorithm to generate the policy. Using the older version of the generated policy from an older model is highly likely to fail. For WebCloak, it is not clear whether the experimental data applies the 2-level defense proposed by the paper.

To make a fair comparison, the paper needs to conduct the experiment using newer LLMs with the Web-IP and WebCloak algorithms instead of only applying the outdated text example from the paper's appendix

  1. There are no concrete examples of injected text, but only the prompts for text generation.
Questions for authors
  1. How do you reproduce the Web-IP and WebCloak experiments?

  2. Can you make a point-to-point comparison between PACT and Web-IP and WebCloak? In this case, we can better understand the contribution of PACT. I suggest you make a table for it.

  3. Can you provide an example of the generated policies?

Reviewer_wj5nOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PACT, a method for optimizing content-use policies that help content owners restrict unwanted LLM use. PACT constructs policies through a hierarchy of protection premises and response directives, then refines them through model-guided evolutionary search using answer suppression as feedback. The policies are placed around protected content without changing the original text. Experiments on question answering and summarization report high suppression rates on unseen content and transfer across models.

Critical strengths
  1. Giving content owners a way to express restrictions on automated use is a relevant problem. The proposed approach supports both whole-document and selected-span protection without requiring access to model parameters.

  2. The reported results demonstrate strong answer suppression in the evaluated settings.

Critical weaknesses
  1. Unclear threat model and insufficient adversarial evaluation. The paper does not clearly define the intended deployment scenarios, trust boundaries, or adversary capabilities. In particular, the adversary’s control over content preprocessing and model instructions, knowledge of the policy, and ability to make repeated adaptive queries remain insufficiently specified. The robustness evaluation in Section 4.4.1 and Appendix B.2 only prepends the fixed instruction “ignore the protection in the document.” This is insufficient to establish robustness against adaptive attempts to bypass a known policy. Without an explicit threat model, it is difficult to judge whether the evaluated attack represents the intended adversary or adequately supports the protection claims.

  2. Limited evidence of practical protection against deliberate circumvention. PACT leaves the protected content readable and relies on the accompanying policy remaining attached and being followed by the model. A third party that obtains the plaintext and controls preprocessing could remove the policy before submitting the content, returning the input to an unprotected setting. Routine content extraction, retrieval chunking, or compression may also separate the policy from the protected information. Although the authors acknowledge that protection depends on the policy remaining attached to the content, they do not explain why this assumption is realistic in the intended deployment setting.

Questions for authors
  1. What is the intended deployment scenario, and who is the adversary? Does the adversary only submit questions to a fixed service, or can they also control content preprocessing, model selection, and system instructions? Please clarify their knowledge and query access.
  2. Under the clarified threat model, can the authors provide stronger adversarial evaluation, including policy-aware iterative prompting and content preprocessing where permitted? Evidence that protection survives these attempts would materially affect my assessment.
2 comments

Dear Reviewer,

Thank you again for your valuable comments. As the discussion period is nearing its end, we would like to kindly ask whether our responses have adequately addressed your concerns. Please let us know if any further clarification is needed. If our responses have resolved your concerns, we would sincerely appreciate your consideration in updating your evaluations accordingly.

Dear Reviewer,

We sincerely appreciate your time and effort in providing such thoughtful reviews and insightful comments.

Could you please kindly let us know if our rebuttal has addressed your concerns?

If you have any further questions regarding our work, we would be delighted to address them.

If our responses have addressed your concerns, we would greatly appreciate your reconsideration of your overall assessment.

Best regards,

Authors

Reviewer_bxeWOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Policy Adaptation for Content ProTection (PACT), a method to prevent unauthorized and unintended use of external content by LLMs by attaching natural language policies. The policy generation is performed in a hierarchical manner where a policy designer LLM generates candidate policies and they are iteratively improved using evolutionary search. The policy is optimized using Answer Suppression Rate, which essentially computes the overlap between generated protected content and reference content.

The authors evaluate PACT on question answering and summarization across four datasets (ContractNLI, Enron, Slack, and MTS-Dialog) and five LLMs. They consider two settings: Global Protection, where the entire document is protected, and Local Protection, where only selected portions are protected. The paper provides several experimental results: near perfect suppression under Global protection, comparison with other baselines, cross model generalization, robustness to conflicting instructions and brief analysis of policy optimization.

Critical strengths
  • The paper is well written.
  • The method achieves strong results, however as noted below this could simply be because of task simplicity.
Critical weaknesses
  • The first issue of the paper is the weak evaluation metric. Answer suppression is essentially a token overlap metric, so I would expect that even paraphrasing would reduce this metric. There should be experiments showing that simple operations like semantically correct paraphrasing does not impact this metric.
  • The task at hand appears extremely simple. In fact, a prompt in Figure 8, Appendix C.3 shows the simplicity of a local protection policy. Coupled with this, since the goal is to prevent the use of unauthorized content, it seems that the main focus of the experiments should be adversarial.
    • The authors evaluate robustness by introducing a fixed instruction asking the model to disregard the protection policy. However, that is highly insufficient to establish robustness under an adversary.
  • One of the baselines used in the paper is Web-IP, which achieves highly successful results in its original experiments. However, Web-IP's suppression rates in PACT's Table 1 are extremely low, ranging from 0.52% to 11.90%. This discrepancy makes the results unreliable. I strongly urge the authors to provide details as to how the baselines were adapted, whether the adoption was fair and if so, provide an explanation as to why the baselines have such low suppression rates.
Questions for authors

Please see above.

2 comments

Dear Reviewer,

Thank you again for your valuable comments. As the discussion period is nearing its end, we would like to kindly ask whether our responses have adequately addressed your concerns. Please let us know if any further clarification is needed. If our responses have resolved your concerns, we would sincerely appreciate your consideration in updating your evaluations accordingly.

Dear Reviewer,

We sincerely appreciate your time and effort in providing such thoughtful reviews and insightful comments.

Could you please kindly let us know if our rebuttal has addressed your concerns?

If you have any further questions regarding our work, we would be delighted to address them.

If our responses have addressed your concerns, we would greatly appreciate your reconsideration of your overall assessment.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.