- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents MemPermit, a write-time governance framework for shared memory in multi-agent LLM systems. The key idea is to jointly decide whether a candidate memory should be persisted and which readers should be allowed to access it. MemPermit combines mandatory risk-triggered specialist verification with a cost-aware sequential verification mechanism based on the expected value of verification (EVoV). It further enforces a non-expanding visibility scope so that future policy changes cannot broaden access beyond the audience determined at write time. Evaluation on three datasets and two models shows reduced access-control violations and memory-injection attack success while keeping benign utility degradation.
Critical strengths
- Shared memory creates security risks that are not fully addressed by either content filtering or read-time access control alone. Treating persistence and audience selection jointly is a sensible formulation.
- The paper considers both improved security and over-refusal and benign memory utility.
- Extensive experiments cover access-control violations, memory injection, benign long-term memory utility, and verification cost, and the results seem sound.
Critical weaknesses
- The framework is quite complex relative to the demonstrated gains and there's no code provided for reproducubility. MemPermit contains lightweight risk modules, RuleGuard, five specialist roles, belief updates, EVoV routing, reassessment, a joint aggregator, quarantine handling, and manually specified asymmetric losses. While each component is motivated individually, it is less clear whether this full complexity is necessary in practice. And a simpler end-to-end or learned write-time policy would be an important comparison.
- The asymmetric loss in Eq. 3 assigns substantially different costs to harm, conflict, unauthorized disclosure, etc. And there is no clear justification regarding why they should be set in this way.
- MemPermit assumes trusted identity and policy services, while compromise of the policy service or storage administrator is explicitly out of scope. This is understandable for scoping the work, but these components are particularly important because the proposed guarantees ultimately depend on their correctness.
- All five specialists within each experimental arm use the same underlying model. The authors acknowledge that their marginal belief updates do not model correlations among same-backbone specialists. In practice, specialists based on the same model may make highly correlated mistakes, potentially making sequential verification appear more informative than it actually is.
Questions for authors
- How sensitive are the results to the risk weights in Eq. 3? Would the advantage of the joint persistence–audience policy remain under substantially different but still reasonable security/utility preferences?
- How much of the full MemPermit pipeline is actually necessary? Could the authors compare against a simpler learned or LLM-based write-time controller that directly predicts {reject, audience} using the same available information and verification budget?
- How correlated are the errors of the five specialists? Since they use the same underlying model, it would be useful to quantify agreement/error correlation and examine whether using heterogeneous models changes the effectiveness of sequential verification.
- How well does the framework scale with the number of principals? The current formulation maintains reader-specific authorization risks and considers audience selection explicitly. It would be useful to discuss or experimentally evaluate settings with much larger and more dynamic reader populations.
Thank you for suggesting a direct LLM controller. We evaluated Direct-LLM on GateMem with GPT-4o-mini. MemPermit Full lowered the access-control violation rate (ACV) from 19.2% to 11.2%, with higher utility and fewer refusals of legitimate requests.
1. How sensitive is the decision to the loss weights?
In Eq. (3), rejecting a benign memory costs 10, retaining harmful content costs 100, and retaining conflicting content costs 20. Let wdis be the cost of unauthorized disclosure and qgood the product of the marginal probabilities that each of five risks is absent: harm, source unreliability, conflict, low retention value, and task mismatch. At qgood=1, with an exclusion cost of 2, including an authorized reader is preferable to excluding them when disclosure risk is below 2/(wdis+2). For disclosure weights of 25, 50 (the paper setting), and 100, this threshold is 7.41%, 3.85%, and 1.96%. For any positive weights and fixed risk estimates, Full minimizes estimated loss over all feasible actions, including Decoupled's chosen action. Table 2 measures the empirical trade-off at the prespecified weights.
2. How does Full compare with a direct controller?
For each candidate, Direct-LLM outputs either rejection or the set of readers who may access the stored memory. Full uses the same backbone and jointly selects admission and audience from risk estimates.
Table R1 (new). GateMem, GPT-4o-mini.
At similar observed cost, Full lowered ACV by 8.0 percentage points (pp), raised utility by 1.5 pp, and lowered over-refusal by 3.1 pp. The runs differed in cost by USD 0.04 per 1,000 candidates; both were below USD 0.50.
Table 2 isolates the decision rule under identical candidates, observations, and final beliefs. Decoupled decides admission from content risk before choosing an audience. Letting audience risk inform admission lowered ACV by 1.23 pp (Full minus Decoupled; two-sided 95% paired episode-cluster bootstrap CI: −2.11 to −0.47 pp), with utility down 0.9 pp and over-refusal up 1.5 pp. Compared with Admission-only, which must admit for all authorized readers or reject, Full lowered ACV by 2.8 pp, raised utility by 1.1 pp, and lowered over-refusal by 2.6 pp. Narrowing the audience lets Full retain a memory that is appropriate for only some readers.
3. What if specialists sharing a backbone make correlated errors?
Specialists judge different risks. A safe source judgment concerns reliability; a safe audience judgment concerns disclosure to a reader. Eq. (6) updates only the reported coordinates. Measuring shared errors requires comparing each role with its own target on the same candidate. Correlated errors could make another call appear more valuable than it is.
A mixed-backbone experiment would change the team. Fig. 4 holds the team fixed and changes routing. Comparing actual cost, ACV, and utility does not require independent specialist errors. For GPT-4o-mini and DeepSeek-V4-Pro, respectively, sequential verification saved 27.4% and 27.1% versus Static Full (all five specialists) and 10.0% and 8.5% versus One-shot. Across these four routing contrasts (sequential minus baseline), the one-sided 95% upper bound on the ACV difference was at most +0.95 pp, and the one-sided 95% lower bound on the utility difference was at least −1.8 pp, within the +1/−3 pp margins.
4. How does the system handle more readers and changing permissions?
Given authorization and per-reader risks, Eq. (3) selects an audience in O(∣A∣) time for the requested reader set A. One scan includes all authorized readers whose inclusion lowers loss. If all increments are nonnegative, the reader with the smallest increment gives the best nonempty audience; we compare its loss with rejection. An empty authorized subset yields rejection. Larger requests also incur per-reader authorization, risk-estimation, and storage costs.
At retrieval, Eq. (8) intersects the stored audience with current authorization. Revocation blocks subsequent reads; later grants do not expand the stored audience. Identity and policy services supply the authenticated reader and current permissions, and the backend enforces the intersection. Missing identity or unresolved policy results in denial.