- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper introduces Priview, a framework for information disclosure in Federated Multi agent LLM based systems. Priview does it in three step process with each step having a clear role. V proposes candidate views from the request, A authorises with the help of owner policy and ledger and finally S gives the response using only the approved view. The framework formalizes this with robust declassification and measure performance in terms of utility and privacy. The paper reports near raw utility scores and zero items leaked on multiple datasets.
Critical strengths
- The problem is well motivated. Introduces a new angle to information disclosure by using robust declassification into LLM disclosure scenarios.
- Design of the solution for prompt injection and differencing attack looks good. A never sees the actual request sent and so no influence on the authorization outcome. S's only private input is the approved view, so it cannot reveal private data beyond it.
Critical weaknesses
- The zero leaks mostly comes from the structure and setup. Views are not generated(fixed for a scenario) and consent is given to Priview as strutured input. Also, auditor uses the same rules that the authorizer used. So, the set does not allow Priview leaking, but the baselines do not have the same setup. They have to infer consent from request itself.
- The auditor used might miss leaks when paraphrases appear, since it uses exact substring matching. The auditor also decides a message was authorized by a marker phrase inside the message text itself
- Weak Baselines in general. CoT+Sens is forced to disclose and ToM is not allowed to tell exact values.
Questions for authors
- Does the zero leak counts change if the auditor determines authorization from the round and fact identity instead of using a marker phrase in the message text. Also, will it change if use some kind of semantic check for catching leaks?
- Can the consent-aware 1-2-3 check looks like fair baseline comparison. Since the same permissions as Priview are given it would perform on similar levels is the expectation. Maybe that can used as a stronger baseline ?
- Can V generate views instead of taking the fixed views from the scenarios? Can it generate the view and rank them? How would it affect A's performance in that case
- ToM baseline tracks history (some kind of scratchpad to track states would be used) and also has zero combination leaks . What does the ledger add for privacy over general ToM methods?
- Can we measure the effectiveness of the compression of the ledger proposed? Is some kind of evaluation of the compression possible?
- Small Nitpicking, but can the model versions used for evaluation be mentioned more clearly?
3 comments
Thank you authors for the detailed comment. The new experiments address most of my concerns and the check+consent comparison is a good addition. But, these results would replace much of the paper's evaluation, and they were run with small run counts, one benchmark family, and an LLM judge as the semantic auditor. So, I will retain my rating as my final score and will increase soundness score for the paper.
Thank you for recognizing the additional evidence and increasing the soundness score. We would like to clarify what the Check+consent comparison establishes.
Check+consent incorporates trusted owner-policy authorization, which is central to our trust-source separation principle. This addition gives the baseline the same owner-permitted information as Priview, rather than requiring it to infer permission from the request. Their comparable performance on short HiddenBench interactions shows that this authorization interface enables useful disclosure in both methods.
The remaining difference appears when individually permitted disclosures accumulate across rounds. The short benchmark interactions do not explicitly stress malicious requests or long disclosure gaps. We therefore added a targeted case with two complementary disclosures separated by 20 discussion rounds. Check+consent leaks the protected combination in 3/3 runs, while Priview has 0/3 leaks. This isolates the role of checking each proposed release against accumulated disclosure state, beyond providing trusted consent.
We will distinguish these targeted mechanism tests from the broader benchmark results in the revision.
The agentic experiment retains task utility and zero detected leaks with generated views. In the supplementary agentic experiment, we replaced V’s fixed-candidate selection with LLM generation of 1–4 candidate disclosures from the request and local evidence, each with a predicted utility score. A then assesses semantic compliance and checks encoded policy rules deterministically. On DeepSeek HiddenBench, this implementation achieves 91.7% accuracy and 48/48 runs without detected leakage.
Baselines using the owner’s privacy policy retain Priview’s safety advantage in the cumulative-disclosure case. We removed forced disclosure from CoT+Sens and the extra exact-value restriction from ToM. Both now use the same owner policy and trusted permissions as Priview. On Qwen HiddenBench, CoT+Sens+consent, ToM, and Priview achieve 60.4%, 72.9%, and 70.8% accuracy. Their zero-leak counts are 42/48, 47/48, and 48/48 under deterministic auditing, and 30/48, 6/48, and 48/48 under semantic auditing, respectively. Check+consent also uses the owner’s privacy policy and matches Priview at 70.8% accuracy and 48/48 deterministic zero-leak runs. The difference between Check+consent and Priview lies in cumulative disclosure: in the case with 20 intervening discussion rounds, Check+consent leaks in 3/3 runs versus 0/3 for Priview.
Semantic auditing detects additional baseline leaks while Priview remains at 48/48 zero-leak runs. A separate DeepSeek post-hoc auditor checks paraphrases, entailment, and cross-round inference, taking a majority vote over three judgments per owner. The Qwen zero-leak counts change from 42/48 to 30/48 for CoT+Sens+consent and from 47/48 to 6/48 for ToM, Priview remains at 48/48. This identifies semantic leaks missed by fragment matching, including in the history-aware ToM baseline.
Priview supports round-specific permission activation without changing the original fact-based authorization conditions. The original datasets specify permissions by fact, so the round-based extension does not change their authorization conditions. We separately constructed a case in which permission takes effect in round 4: Priview withholds the protected information in rounds 1–3 and discloses it in round 4. Semantic fact auditing addresses a different question—what protected information the recipient can infer. As reported above, this stricter audit detects additional baseline leaks while Priview remains at 48/48 zero-leak runs.
The ledger reduces authorization-input characters by 96.8% in the stress case. In the 20-round stress case, supplying full history to A yields 2/3 leaks, versus 0/3 with the ledger. Mean authorization input falls from 10,574.7 to 337.0 characters, a 96.8% reduction. This directly measures the ledger's compact representation in addition to retaining history.
The revision will specify exact model versions and evaluation settings. The experiments use DeepSeek-v4-flash, Qwen-max, and GPT-5.6-Sol backbones. We will specify their exact model versions, run dates, and judge settings in the revised experimental setup.