- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes PrivMeSA, a privacy-aware multi-agent framework for clinical decision support that enables a local LLM to consult more capable remote specialist agents while minimizing patient information disclosure.
The framework combines two key components. First, a privacy-aware reinforcement learning objective trains the local agent to balance clinical task accuracy, direct disclosure of personal information, and re-identification risk measured through registry-based k-anonymity over multi-turn consultation transcripts. Second, a self-evolving local lesson memory distills completed remote consultations into reusable guidance, allowing subsequent cases to benefit from previously acquired expertise without additional remote communication.
The authors introduce a multi-turn emergency-department benchmark constructed from MIMIC-IV-ED, including repeated patient visits and specialist-initiated follow-up questions. Experiments with Gemma 4 12B and Granite 4.2 8B demonstrate improved task accuracy and substantially reduced disclosure compared with delegation and PAPILLON baselines. Additional ablations investigate the training objective, consultation behavior, and benefits of test-time memory accumulation.
Critical strengths
1. Important and well-motivated problem formulation for privacy-aware multi-agent collaboration.
The paper extends privacy-conscious delegation beyond single-message sanitization by explicitly considering the accumulation of quasi-identifiers across multi-turn consultations and repeated patient visits. Evaluating disclosure over complete outbound transcripts and linking patient encounters provides a more realistic privacy assessment than examining individual messages in isolation. The formulation is particularly relevant to healthcare applications where local systems may need assistance from remote models without exposing sensitive patient information.
2. Strong empirical results and meaningful analysis of privacy-utility trade-offs.
The proposed approach achieves a favorable balance between task accuracy and privacy across two local model backbones. The improvements are especially pronounced for Granite 4.2 8B, where mean accuracy increases from 0.425 under delegation to 0.583, while direct disclosure falls from 98.0% to 0.2%. The paper also includes useful ablations of privacy-aware training and consultation behavior. The test-time memory experiments provide evidence that previously acquired knowledge can reduce remote interactions without degrading predictive accuracy in the evaluated setting.
Critical weaknesses
1. The privacy guarantees are limited by the attacker-dependent evaluation protocol.
Although the paper reports that no patient is re-identifiable under PrivMeSA, this conclusion is based on a particular registry-matching procedure and an LLM-based quasi-identifier extractor. Appendix B.2 states that visits are excluded from re-identification statistics when extraction errors produce attributes inconsistent with the patient's actual record. This exclusion rule could lead to optimistic estimates of privacy, particularly if extraction errors differ across methods.
Furthermore, the evaluation focuses on a predefined set of quasi-identifiers and uses k ≤ 10 as the re-identification threshold. While the comparison with an independent attacker backbone is helpful, it does not establish robustness to broader auxiliary information, different linkage strategies, or alternative attacker models. The reported zero re-identification rate should therefore be interpreted as a result under the specified evaluation protocol rather than a general privacy guarantee.
2. The experimental comparisons do not fully isolate the contributions of privacy-aware RL and self-evolving memory.
The main baselines, Delegation and PAPILLON, use untrained local models, whereas PrivMeSA benefits from task-specific GRPO training and an accumulated lesson memory. This makes it difficult to determine how much of the accuracy improvement arises from privacy-aware optimization versus general task-specific fine-tuning or memory reuse.
The local-only RL baseline and the reward ablations partially address this issue, but the reward ablations use a simplified configuration with a single remote specialist and shorter training. A more direct comparison using an accuracy-only trained delegation policy with identical specialist access and training budget would better establish the contribution of the privacy-aware objective.
Additionally, the test-time memory experiment reports a reduction of approximately 15% in remote messages, but the associated accuracy gains are relatively small and vary across evaluation quarters. More detailed analysis of memory retrieval correctness, lesson quality, and failure cases would strengthen the claim that the memory safely transfers expertise across patients.
Questions for authors
Q1. Robustness of re-identification evaluation: Appendix B.2 excludes visits when the attacker extracts attributes that do not match the patient's record. What fraction of visits is excluded for each method, and how would the privacy results change if these visits were retained or treated conservatively? Could the authors additionally evaluate robustness to alternative quasi-identifier extraction and matching strategies?
Q2. Attribution of gains from privacy-aware RL: Could the authors compare PrivMeSA against an accuracy-only GRPO-trained delegation baseline with the same local model, remote specialists, training budget, and memory configuration? This would help isolate the benefits of the privacy-aware reward from those of task-specific policy optimization.
Q3. Memory quality and potential negative transfer: How often does the lesson memory retrieve clinically incorrect or inapplicable guidance, and how frequently does the local policy accept such lessons? Could the authors analyze whether memory accumulation introduces errors or negative transfer across clinically similar but distinct patient cases?
Q4. Generalization of privacy behavior: How well does the learned disclosure policy generalize to different remote specialists, different follow-up-question strategies, and unseen quasi-identifier combinations? The current results demonstrate strong performance within the constructed consultation environment, but the robustness of the policy to changes in the interaction and attacker distributions remains unclear.
3 comments
Response to Reviewer WxRH (part 2 of 2)
W2, Q3. Quality of memory lessons and negative transfer
On the 897-case test stream, the memory offered 419 lessons, and the policy answered 384 of these tasks (92%) from the memory lesson instead of asking a remote specialist. We have added an LLM judge (GPT-5.6-Luna) to rate each lesson. We have also compared each answer with the answer to the same task in an empty-memory run, where the policy had to ask a remote specialist.
Table R4: Accuracy of answers taken from memory, against asking a remote specialist for the same task (Gemma 4 12B, 897-case test stream).
| Lesson type | Share of the 384 | Accuracy from the lesson ↑ | Accuracy asking a remote specialist ↑ | Difference (points) |
|---|---|---|---|---|
| All | 100% | 0.591 | 0.612 | −2.1 |
| From same patient, earlier visit | 17% | 0.500 | 0.516 | −1.6 |
| From another patient | 76% | 0.608 | 0.632 | −2.4 |
| Judged clinically unsound | 4% | 0.267 | 0.400 | −13.3 |
- The judge rated 64% of used lessons a full fit to the patient, 36% a partial fit and none inapplicable. Clinically unsound lessons are rare (15 of 384).
- Answering from a lesson is only slightly less accurate than asking a remote specialist. Over the whole stream, memory keeps accuracy similar to an empty memory (0.591 vs 0.588) with 15% fewer messages.
- Lessons from other patients lose about as much as those from the same patient (−2.4 vs −1.6 points), so we see no negative transfer between similar but distinct patients.
Q4. Generalization to other specialists and follow-up strategies
Different remote specialists
We have replaced GPT-5.6-Luna with Claude Haiku 5.5 as the remote specialists, without retraining.
Table R5: Results with Claude Haiku 5.5 as the remote specialists (local model Gemma 4 12B, 453 cases).
| Method | Mean task accuracy ↑ | Direct disclosure (any leak) ↓ | Re-identification: k ≤ 10 ↓ | Re-identification (linked visits): k ≤ 10 ↓ |
|---|---|---|---|---|
| Delegation | 0.542 | 88.7% | 44.4% | 91.0% |
| PAPILLON | 0.493 | 19.9% | 1.8% | 11.0% |
| PrivMeSA (ours) | 0.553 | 1.1% | 0.0% | 1.0% |
- Every method loses 2 to 4 accuracy points compared with GPT-5.6-Luna (Table 1).
- PrivMeSA stays the most accurate and most private, with almost no change in direct disclosure (1.1% vs 0.9%).
Different follow-up questions
We have tested a new follow-up strategy, where the remote specialists ask for personal details with new reasons PrivMeSA has not seen in training. For example, the diagnostician asked for the full name, address and next of kin "because the record cannot be filed without them", and the clinician asked for the job title and employer for "return-to-work planning". PrivMeSA declined all 1,096 new requests without disclosing anything.
Unseen patients and quasi-identifier combinations
All test patients are held out from training (Sec. 4.1), so every result above is on quasi-identifier combinations the policy has never seen. We have also evaluated PrivMeSA without retraining on MC-MED (Kansal et al., 2025): 448 cases (100 patients) from the Stanford emergency department, a different hospital than MIMIC-IV-ED. Its registry holds all 70,545 MC-MED patients.
Table R6: MC-MED results (448 cases, Gemma 4 12B, no retraining). Task columns give accuracy.
| Method | Mean task accuracy ↑ | Disposition ↑ | Primary diagnosis ↑ | Direct disclosure (any leak) ↓ | Re-identification: k ≤ 10 ↓ | Re-identification (linked visits): k ≤ 10 ↓ |
|---|---|---|---|---|---|---|
| Delegation | 0.516 | 0.685 | 0.347 | 93.5% | 78.4% | 98.0% |
| PAPILLON | 0.482 | 0.681 | 0.284 | 19.4% | 6.1% | 24.0% |
| PrivMeSA (ours) | 0.497 | 0.679 | 0.315 | 1.8% | 1.1% | 15.0% |
- PrivMeSA has the lowest disclosure and re-identification at the new hospital, without retraining.
- Re-identification is higher than on MIMIC for every method, because the registry is a third the size.
- PrivMeSA's mean accuracy is within 2 points of Delegation and above PAPILLON.
We hope these results address your concerns, and we are glad to answer any further questions.
Thank you again for your detailed review and your positive assessment of our work. We have uploaded a revised paper, with all new results from the discussion in blue.
As promised in our reply, the paper now reports re-identification at every threshold from k ≤ 1 to k ≤ 100 (Appendix B.5) and states that our re-identification results hold for our registry and attacker rather than as a formal guarantee (Section 4.2). It also adds the excluded visits (Appendix B.4), the accuracy-only GRPO run (Table 4 and Appendix E.2), MC-MED (Appendix A.4), lesson quality (Appendix E.6), and the results with Claude Haiku 5.5 and new requests (Appendix E.7).
If you have any remaining concerns or questions, please let us know and we will try our best to answer and address them in the remaining time. If our responses have addressed your concerns, we would be grateful if you would consider raising your score.
Response to Reviewer WxRH (part 1 of 2)
Thank you for the detailed review and for finding our multi-turn, linked-visit evaluation more realistic. We have run all four requested analyses and address them below, in two parts.
W1, Q1. Robustness of re-identification to the attacker's protocol
Excluded visits (Appendix B.2), kept or treated conservatively
A visit is excluded when an extracted value contradicts the patient's record, so the true patient is not among the k matches. As requested, we have measured how many visits are excluded and how the result changes when they are kept or counted as re-identified.
Table R1: Share of visits re-identified (k ≤ 10) under three ways of handling excluded visits, where the true patient is not among the attacker's k matches. Dropped: excluded visits are removed, as in the paper. Scored by k: excluded visits stay in and count as re-identified only if k ≤ 10. All re-identified: every excluded visit counts as re-identified, the most conservative choice.
Alternative extraction and matching strategies
Our attacker counts only explicitly stated values, so every match rests on information that was actually sent. We have added an extended attacker that also reads ranges and descriptive terms ("in her 70s", "elderly", "febrile") and matches the chief complaint by meaning, with a threshold set on Delegation only. PrivMeSA was not trained against it.
Table R2: Accuracy and re-identification under the extended attacker, counting only matches that include the true patient.
W2, Q2. Accuracy-only GRPO with the same specialists, budget and memory
Appendix E.2 used a shorter run with one remote specialist. As requested, we have now trained the full PrivMeSA system with both privacy weights at 0 and nothing else changed: the same local model, three remote specialists, memory and 200 steps.
Table R3: Accuracy-only GRPO against PrivMeSA, same setup (Gemma 4 12B, 453 cases). Changes from PrivMeSA are in points.