- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper builds on Zhang & Yang (2026), which alternately searches for more effective attacker and defender instructions in multi-agent privacy simulations using an LLM as prompt optimizer. PRISM replaces the step that decides which instruction each parallel thread revises next. While the original work uses non-transparent LLM reflection and rewrite, PRISM embeds instruction text, learns a sparse dictionary, fits a closed-form Bayesian surrogate of leak velocity. On the attack side, it selects seeds via GP-BUCB. On the defense side, it derives an LP-solvable zero-sum game. In settings with matched simulation and optimizer-call budgets, attacks transferred from PRISM’s search reach higher leak velocity on the 100 held-out test scenarios than the baseline.
Critical strengths
-
The paper is well-written and easy to follow. It optimizes over a previous search algorithm and compares with the baseline at matched budget in shared environment. It is also very honest and transparent about negative results, showing that their optimization over the original algorithm does not demonstrate a clear improvement in sample efficiency.
-
The paper points out a problem in the search algorithm of Zhang & Yang (2026), which selects the attack / defender instruction with highest / lowest leak velocity. The paper empirically demonstrates that the selection often reduces to random choice in actual run, because there are many instructions with same leak scores, which is a useful observation.
Critical weaknesses
-
The contribution seems incremental on prior work. PRISM keeps the existing simulation and most of the instruction optimization pipeline of Zhang & Yang (2026), and changes which instruction will be selected for further rewrites. While the leak velocity improvement on Testing-100 is non-trivial, there seems to be no broader benefits of the proposed mechanism (e.g. sample efficiency, simulation costs).
-
The paper does not compare itself with other existing prompt search / prompt optimization baselines. The only baseline in paper is even inconsistent with the original implementation of Zhang & Yang (2026), since in their setup the LLM optimizer carries history. Therefore, it's hard to say whether the attacker side's gain comes from their modelling or preserving certain history states.
-
The practical value of the defense side remains to be proved. While the paper discusses their defense strategies in detail, it's not compared with the baseline defense on a live run, making it hard to see its value. An informative online evaluation is needed to demonstrate the construction’s practical benefit.
Questions for authors
None