acceptodds
Withdrawn from ICLR 2027

PRISM: Transparent Strategy Search for Privacy Risks in LLM Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Discovering privacy attacks on LLM agents by search is expensive: each candidate attacker instruction must be evaluated in a simulated multi-agent conversation, and the search framework of Zhang & Yang (2026) decides what to try next inside one reflect-and-rewrite LLM call, with no explicit representation of tactics and no surrogate-based rule for choosing among proposals before they are simulated. PRISM is a drop-in replacement for that proposal step. It represents each instruction as a sparse code over named tactic primitives learned from its text, fits a closed-form Bayesian surrogate of leak velocity (the benchmark’s metric; higher means a stronger attack), and uses the surrogate to choose which previously evaluated instructions the parallel search threads revise next; the rewrite itself remains the baseline’s own LLM call, and the simulator, judge, metric, data splits and per-step simulation and optimizer-call budgets are unchanged. On the benchmark’s 100 held-out scenarios, run entirely on open-weight models, the attack instructions transferred from PRISM’s search reach a mean leak velocity of 0.277 against 0.188 for the baseline’s (paired difference +0.089, 95% bootstrap interval [+0.025, +0.153] over scenarios). We also measure a property of the benchmark’s selection rule, which breaks ties among equally scored candidates uniformly at random: from a defended starting point in our environment all 30 candidates had zero observed leakage in 87 of 100 steps, so the observed scores did not distinguish candidates within those steps. A game-theoretic defense construction is derived and evaluated offline on released trajectories; the comparative defense searches stopped before it could activate, so no live comparison of defense proposers is reported.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
LLM agents, privacy leakage, automated red teaming, Bayesian optimization, interpretable representations, sparse coding, multi-agent systems

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cA9FOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper builds on Zhang & Yang (2026), which alternately searches for more effective attacker and defender instructions in multi-agent privacy simulations using an LLM as prompt optimizer. PRISM replaces the step that decides which instruction each parallel thread revises next. While the original work uses non-transparent LLM reflection and rewrite, PRISM embeds instruction text, learns a sparse dictionary, fits a closed-form Bayesian surrogate of leak velocity. On the attack side, it selects seeds via GP-BUCB. On the defense side, it derives an LP-solvable zero-sum game. In settings with matched simulation and optimizer-call budgets, attacks transferred from PRISM’s search reach higher leak velocity on the 100 held-out test scenarios than the baseline.

Critical strengths
  1. The paper is well-written and easy to follow. It optimizes over a previous search algorithm and compares with the baseline at matched budget in shared environment. It is also very honest and transparent about negative results, showing that their optimization over the original algorithm does not demonstrate a clear improvement in sample efficiency.

  2. The paper points out a problem in the search algorithm of Zhang & Yang (2026), which selects the attack / defender instruction with highest / lowest leak velocity. The paper empirically demonstrates that the selection often reduces to random choice in actual run, because there are many instructions with same leak scores, which is a useful observation.

Critical weaknesses
  1. The contribution seems incremental on prior work. PRISM keeps the existing simulation and most of the instruction optimization pipeline of Zhang & Yang (2026), and changes which instruction will be selected for further rewrites. While the leak velocity improvement on Testing-100 is non-trivial, there seems to be no broader benefits of the proposed mechanism (e.g. sample efficiency, simulation costs).

  2. The paper does not compare itself with other existing prompt search / prompt optimization baselines. The only baseline in paper is even inconsistent with the original implementation of Zhang & Yang (2026), since in their setup the LLM optimizer carries history. Therefore, it's hard to say whether the attacker side's gain comes from their modelling or preserving certain history states.

  3. The practical value of the defense side remains to be proved. While the paper discusses their defense strategies in detail, it's not compared with the baseline defense on a live run, making it hard to see its value. An informative online evaluation is needed to demonstrate the construction’s practical benefit.

Questions for authors

None

Reviewer_yaenOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper preposes PRISM, a framework for privacy attack search against LLM agents that incorporates explicit strategy representations and predictive guidance. PRISM combines sparse tactical representations, a Bayesian surrogate model, and Upper Confidence Bound (UCB) selection to identify promising attack seeds. The paper also formulates a defense game and derives a conditional upper bound on information leakage under assumptions about surrogate-model error. Across 100 transfer evaluation scenarios, PRISM achieves an average leakage rate of 0.277, outperforming the baseline rate of 0.188.

Critical strengths
  1. The paper explicitly models the problem of deciding which historical attacks are worth further rewriting, addressing the allocation of search resources in expensive simulations. This is a practically relevant and valuable direction.

  2. PRISM outperforms the baseline across 100 transfer evaluation scenarios (0.277 vs. 0.188), providing positive empirical evidence of its effectiveness.

Critical weaknesses
  1. In the experiments, the baseline uses the --no-history setting, which prevents instruction text from being retained across steps. In contrast, PRISM provides the rewriter with the selected seed text and its historical trajectory through an additional seed section. Thus, the two methods differ not only in their seed selection mechanisms but also in the amount of cross-step information available to the rewriter. The observed performance gains may therefore be partly attributable to the additional historical information rather than the Bayesian surrogate model and UCB selection mechanism themselves. In my opinion, the paper does not sufficiently validate the claimed contribution of its search mechanism.

  2. Although the attack agent does not initially have access to sensitive data at the start of the simulation, the LLM for optimizing attack instructions can access the complete trajectories of previous simulations, including sensitive values that have already been leaked, and may incorporate these values into subsequent attack instructions. Meanwhile, the current evaluation metric counts both the defense agent's initial disclosure of sensitive values and its repetition or confirmation of sensitive values already provided by the attacker as privacy leakage. While confirming sensitive information may itself pose privacy risks, these two behaviors reflect different attack capabilities. I think the current results may not clearly distinguish between the attack method's ability to discover previously unknown secrets and its ability to induce the model to confirm already known information.

  3. In the main experiments, the original defense search quickly achieved zero observed leakage and terminated before PRISM's defense mechanism met its cold-start requirement. Hence the paper does not provide a meaningful online comparison between PRISM's defense algorithm and the baseline. Furthermore, the leakage upper bound in Theorem 3 relies on an unverified assumption of uniformly bounded surrogate-model approximation error over the entire convex hull of attack embeddings. In the online sanity-check run, the computed empirical upper bound is approximately 1.90, exceeding the maximum possible leakage score of 1 and thus providing no meaningful leakage guarantee. Therefore, the paper currently lacks sufficient evidence that the proposed method can generate defense instructions that provide reliable privacy protection.

  4. The paper repeatedly discusses limitations without adequately aligning its main contribution claims with the available evidence. In particular, the defense-side method is presented as a central contribution despite the lack of a meaningful online evaluation.

Questions for authors
  1. Appendix D reports a within-step Spearman correlation of only 0.046 between the observed leak velocity of selected seeds and that of their rewritten descendants, and a correlation of −0.001 between the surrogate-predicted mean and descendant leak velocity. PRISM uses surrogate-based UCB scores to select seeds for further rewriting, could the authors explain these near-zero correlations? If seed scores do not reliably predict descendant performance, what direct evidence demonstrates that surrogate-guided seed selection outperforms random selection or simpler strategies based on historical observed leak velocity?

  2. Could the authors provide additional online comparisons in which PRISM’s defense mechanism is actually activated, evaluating both privacy leakage and helpfulness? Furthermore, since Theorem 3 relies on an unverified uniform approximation-error assumption, could the authors clarify the conditions under which this assumption holds or provide a non-vacuous error bound with statistical guarantees?

Reviewer_BgzkOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces PRISM, a proposal engine for the alternating attack–defense search over privacy-leaking instructions in LLM agents.

Critical strengths

The proposal engine is inspectable: sparse codes over named tactic families, closed-form surrogate coefficients, acquisition scores, and per-step decisions are logged, which is a meaningful improvement over opaque prompt-proposal baselines.

Critical weaknesses
  1. The abstract should be self-contained and should not contain literature citations, theorem numbers, appendix references, or figure references.

  2. The defense bound in Theorem 3 rests on an approximation-error hypothesis that is not verified. The empirical allowance is a heuristic without simultaneous or held-out coverage guarantees, and the sanity runs produced a vacuous bound. The offline leave-one-defense-out check covers nine of ten folds, but it is not the registered endpoint and, read by the pre-registered rule, its one exceedance would count as disconfirming.

  3. The evaluation is heavily shaped by the local open-weight environment, which is harder to attack than the published environment. Absolute numbers are not comparable across environments, and the paper says so.

  4. The presentation is dense and caveat-heavy. The main text often reads like a sequence of qualifications.

Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.