acceptodds
Under review as a conference paper at ICLR 2027

LatentDx: Latent Multi-Agent Communication for Cross-Hospital Rare-Disease Diagnosis

Abstract

Rare diseases affect approximately million patients across more than conditions, yet no single hospital encounters enough cases of any one condition for reliable diagnosis. Cross-hospital collaboration could help by allowing a diagnosing institution to use distributed, case-specific diagnostic evidence, but privacy regulations restrict the transmission of identifiable clinical text across institutional boundaries. This setting raises two challenges: existing medical agent systems often rely on textual evidence exchange, while raw latent states such as hidden states and KV caches may still reveal prompt-derived clinical content. We introduce LatentDx a latent multi-agent communication framework in which hospital agents keep private clinical records and retrieved cases local, and send compact latent representations to a host agent for rare-disease diagnosis. In the same-backbone setting, LatentDx-H transmits distilled KV blocks. In the cross-family setting, LetentDx-X transmits compact hidden-state sequences for projection into host input embeddings. On CrossRare-Bench, a self-built large-scale rare-disease benchmark with hospital-level partitions, LatentDx supports strong cross-hospital diagnostic performance, with evaluations in both settings also showing reduced reconstructable clinical content relative to raw-latent communication baselines. Code and data are available at https://anonymous.4open.science/r/LatentDx/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-Agent, Latent Communication, Rare-Disease Diagnosis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5ymKOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies how hospitals can work together on rare-disease diagnosis when patient text cannot be shared. The authors argue that sending text is not allowed, and that sending raw model states can still leak private data. The main challenge is to pass useful diagnostic information between hospitals while limiting this leakage, including when hospitals use different LLMs. The authors propose to send a short, trained latent block instead of text or raw states, and they build a benchmark that measures both accuracy and leakage. They report accuracy close to text sharing, with much lower leakage than sharing raw model states.

Critical strengths

S1. The method keeps most of the accuracy of text sharing while leaking much less.

  • Accuracy is only slightly below a trained host that reads the records directly (Table 21).
  • Leakage is far lower than when raw KV caches are sent, under all five attacks (Tables 3 and 29).
  • A control with the same training and size leaks much more, so the gain is not just from compression (Table 22).

Why this matters to my score: This is the core result and the main reason I lean towards acceptance.

S2. The experiments are thorough.

  • Each attack has a control where nothing is sent (Table 3).
  • Trained controls show that evidence from other hospitals helps (Table 23).
  • The results hold across settings and LLM families, at a small extra cost (App. C–D, Table 2).

Why this matters to my score: These controls make the results believable, so I see the weaknesses below as fixable.

Critical weaknesses

W1. The privacy claims go further than the attacks support.

  • The blocks can be recomputed from known inputs and public weights, so an attacker could check a candidate record directly. This attack is not tested (Sec. 3.1, App. F.3).
  • Only a passive listener is studied. A host or hospital that sends its own queries is not, and the query itself is sent in plain text (Sec. 3.1).
  • There is no noise or differential privacy, and under learned attacks leakage is still above the level seen when nothing is sent (Sec. 4.5, Table 3).

Why this matters to my score: Privacy is half of the contribution. If the simple attack works, the membership result fails. This is why my score is 3, not 4.

W2. Part of the accuracy gain comes from training, and the benchmark is easier than the real setting.

  • All test diseases appear in training, but the main baselines are untrained. On Llama, a trained model that sees only the query already beats them; on Qwen, LatentDx shows no clear gain (Tables 1 and 23).
  • Two of the data sources probably share cases, and duplicates are removed only by exact match (App. A.1).
  • The module is trained on the hospitals' "private" records, and each hospital already covers almost all diseases (Sec. 3.2, Table 5).

Why this matters to my score: It is unclear how much of the gain comes from the method, and whether it would hold in real hospitals.

Questions for authors
  1. In the membership test, is the query case removed from retrieval? Is any part of block construction random? How well does the "compute and compare" attack in W1 work?
  2. Why is it acceptable to send the query's phenotypes in plain text but not the retrieved records? How would you handle a host that sends its own queries to probe the hospitals?
  3. How do baselines that share only a disease label or a local diagnosis perform (for example, Struct. RAG + LoRA), and what do they leak?
  4. How well does the method work on diseases that are in the hospitals' records but were not used to train the module?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 5ymK [1/2]

We sincerely thank the reviewer for the careful review and for recognizing the utility–leakage result as the main reason for leaning towards acceptance. We also appreciate the reviewer's clear articulation of which concerns matter most for the assessment, in particular the scope of our privacy claims. We took this feedback seriously and address it first below. New experiments are marked (new) below and in blue in the revised PDF.

W1 & Q1: Scope of the Privacy Claims and the Compute-and-Compare Attack

As the reviewer identified the scope of our privacy claims as the key remaining issue, we implemented the proposed attack, developed a differentially private variant, and now state precisely which claims hold under which attacker.

Answers to Q1. (1) The candidate is not removed from retrieval. By design, the membership attack queries with the candidate's own phenotypes, so a member hospital retrieves the candidate itself, which favors the attacker. (2) Block construction involves no randomness. Retrieval, the frozen backbone and the distiller are deterministic at inference, so the same query always yields the same block. (3) As the reviewer anticipated, the compute-and-compare attack is effective. With all weights and the complete candidate record, it reaches a TPR of 1.00 at 1% FPR.

What this attack shows. We agree with the reviewer. Against an attacker who holds a candidate's complete record and all weights, the deterministic block does not hide membership, as is also the case for Raw KV and LatentMAS. The attack verifies a record the attacker already has rather than recovering unknown content. With 90% of the candidate's phenotypes its TPR falls to 0.43, and with 70% it falls to 0. Our reconstruction, attribute and inversion results, which measure the recovery of unknown content, therefore stand, while the membership result does not hold against this attacker. We have revised the threat model to state that the attacker may know all weights, including the interface, and qualified the membership row of Table 4 accordingly.

Protecting membership with LatentDx-DP (new). To protect membership against such an attacker, and to address the reviewer's point on noise and differential privacy, we developed LatentDx-DP (Appendix F.5). Each hospital still sends a latent block of the same size, and the host LLM is unchanged. Only the distiller's input changes, and the distiller is retrained on such inputs with the same recipe. Instead of the raw record, the hospital encodes a differentially private summary of its local evidence, released with the analytic Gaussian mechanism (Balle & Wang, 2018). Each query satisfies (4.1,10−5)(4.1, 10^{-5})-DP with respect to adding or removing one record, and the block is post-processing, so the guarantee holds even against an attacker with all weights. The table below summarizes membership leakage under both attackers on Llama-3.2-3B (TPR at 1% FPR, chance is 0.01).

Raw KVLatentDxLatentDx-DP
Learned attacker (Table 4)1.000.03--
Compute-and-compare, complete record1.001.000.02
Accuracy0.350.730.72

LatentDx-DP reduces compute-and-compare membership to near chance at a cost of 0.01 accuracy.

Residual leakage under learned attacks. Beyond membership, the reviewer also noted that leakage under the learned attacks remains above the no-transmission control. This residual is concentrated in disease-level information that the diagnostic task requires, while the inversion decoder's recovery of the retrieved record's phenotype terms stays close to the control (HPO micro-F1 0.27 vs. 0.23 on Llama-3.2-3B, Table 37).

Q2: Plaintext Query and Actively Probing Hosts

We thank the reviewer for this question.

Plaintext query. The query phenotypes belong to the patient of the diagnosing hospital that initiates the consultation, whereas LatentDx protects the historical records held by the responding hospitals. As stated in Section 3.1, we assume that sharing the query is authorized, as in a standard inter-institutional consultation.

Actively probing hosts. A host that issues its own probing queries is the setting of our membership attacks, in which the attacker chooses the candidate's phenotypes as the query (Table 4 and Appendix F.5). Without the complete candidate record, even the compute-and-compare attack weakens sharply (see W1). With the exact record, LatentDx-DP limits what each query reveals about any single record to (4.1,10−5)(4.1, 10^{-5})-DP (Appendix F.5). Since the guarantee is per query, it composes over the queries a hospital answers, which each hospital can bound with a query budget.

AuthorsOpenReview ↗

Response to Reviewer 5ymK [2/2]

W2: Source of the Gain and Benchmark Realism

We thank the reviewer for this careful analysis and address its three points in turn.

Role of training. We compare LatentDx with trained query-only controls that receive the same supervision but no hospital evidence, and the revision brings these trained comparisons into the main text (Section 4.4, Table 3). Removing the evidence lowers accuracy from 0.73 to 0.64 on Llama and from 0.64 to 0.41 on Qwen (Appendix E.2), so the hospital evidence adds substantially on top of training. To also match the latent budget, in the revision we train a query-only model that sends three blocks of the same size without retrieved cases. Averaged over three training seeds, LatentDx exceeds it by 0.07 accuracy and is better on all three metrics (Appendix E.2, Table 30). Trained references that read the shared evidence directly are discussed in our response to Q3.

Near-duplicates across sources (new). Appendix A.1 now reports a near-duplicate audit. No test case shares its HPO set with a training case of the same disease, and 11 of the 401 test cases (2.7%) have one with HPO Jaccard similarity of at least 0.9. Removing these 11 cases changes the accuracy of every method by at most one point. Likewise, when hospitals never return a case with Jaccard of at least 0.9 (or 0.8) to the query, the accuracy of LatentDx changes by less than one point.

Disease coverage across hospitals (new). In the revision each source cohort becomes its own hospital (Appendix D.4). The three hospitals differ in size (5,894 / 804 / 522 cases), disease coverage (219 / 38 / 64 diseases) and annotation practice. LatentDx is best on every cohort and metric, with 0.76 overall accuracy against 0.47 for the strongest baseline. The disease-skewed partition already in the original Appendix D.3 concentrates each disease in one hospital, which holds on average 65% of its cases, and LatentDx keeps the highest accuracy there on both backbones.

Records never used in training (new). We retrain the distiller without any Zenodo record, removing them from the training queries, the validation queries and the training-time hospital databases. Zenodo then joins as its own hospital only at test time. LatentDx still has the highest accuracy of all methods, both on the Zenodo patients (0.55 vs. 0.41 for the strongest baseline) and over all patients (Appendix D.4, Table 21). Records that were never used to train the module therefore remain usable at test time.

Q3: Baselines Sharing Only a Disease Label or a Local Diagnosis

We thank the reviewer for this suggestion. Struct. RAG already shares only each hospital's nearest-case diagnosis and its similarity score, which amounts to a local nearest-neighbor diagnosis (0.42 accuracy on Llama, Table 1).

Utility with host tuning (new). As suggested, we trained Struct. RAG with host LoRA (Appendix D.9, Table 27). It reaches 0.74 accuracy against 0.73 for LatentDx, with a lower macro F1, while tuning 24.3M parameters of the host LLM itself, whereas LatentDx keeps every backbone frozen and trains only its 9.45M-parameter distiller.

What it leaks. Its utility is therefore comparable, but it sends each retrieved patient's diagnosis in plaintext, so any observer of the messages reads it. From LatentDx's blocks, the inversion decoder recovers the retrieved disease in 38% of cases on Llama, against 27% when nothing is transmitted (Table 4).

Q4: Records and Diseases Unseen in Training

We thank the reviewer for this question and conducted additional experiments to answer it. We distinguish records that were not used in training from diseases that were absent from training altogether.

Records unseen in training (new). In the setting of W2, Zenodo is held out from training entirely. No Zenodo record is used as a training query or stored in the training-time hospital databases, and Zenodo joins as its own hospital only at test time. LatentDx still keeps the highest accuracy, both on the Zenodo patients and over all patients (Appendix D.4, Table 21).

Diseases unseen in training (new). We further built a strict split (Appendix D.6) in which 58 diseases are removed from all training queries and training-time hospital databases, while the test-time databases still contain them. We find this setting genuinely challenging, since the interface never saw these diseases during training. To address it, we add name-anonymized copies of half of the training queries, in which every disease name is consistently replaced by an unrelated one so that the answer must be read from the evidence rather than recalled. With this change, LatentDx reaches 0.47 accuracy on unseen diseases, nearly double its original accuracy, while accuracy on seen diseases remains high (Table 23).

Reviewer_RQojOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LatentDx, a privacy-aware multi-agent framework for cross-hospital rare-disease diagnosis. It enables hospital agents to share compact latent representations instead of private clinical records. Two variants are introduced: LatentDx-H for same-backbone communication through KV distillation and LatentDx-X for cross-family communication through latent alignment. Experiments on CrossRare-Bench demonstrate improvements in diagnostic accuracy while reducing recoverable clinical information under evaluated privacy attacks.

Critical strengths
  1. The paper addresses the challenges in cross-hospital rare-disease diagnosis by enabling collaborative reasoning without directly sharing private clinical records.
  2. The experiments cover multiple LLM backbones, cross-family collaboration, non-IID hospital partitions, and various privacy attacks, demonstrating the effectiveness of the proposed latent communication framework.
Critical weaknesses
  1. Lack of clinically verifiable diagnostic evidence. Although LatentDx demonstrates that compact latent representations can support accurate diagnosis, the host agent cannot directly inspect the underlying clinical evidence retrieved from remote hospitals. The framework outputs only a disease prediction without providing verifiable diagnostic rationales or evidence attribution. This limitation is particularly concerning for rare-disease diagnosis, where clinicians often have limited prior experience and require transparent evidence to assess uncertain predictions.
  2. Insufficient comparison with existing medical multi-agent approaches. Although the paper discusses several medical multi-agent frameworks for rare-disease diagnosis, such as RareAgents [1], the experiments mainly compare against simplified text-based collaboration and latent communication baselines. More competitive medical multi-agent methods are not evaluated. Furthermore, the evaluation primarily focuses on relatively small LLM backbones, leaving the generalizability to stronger and larger LLMs insufficiently explored.

References
[1] RareAgents: Autonomous Multi-disciplinary Team for Rare Disease Diagnosis and Treatment, AAAI, 2026.

Questions for authors
  1. How can clinicians verify the diagnostic evidence underlying predictions generated from latent representations? Can the authors provide evidence attribution or clinically validated explanations while preserving patient privacy?
  2. Can the authors compare LatentDx with representative medical multi-agent frameworks, such as RareAgents [1], under consistent datasets and evaluation settings? Such comparisons would help establish the diagnostic advantages of LatentDx beyond its privacy-preserving communication mechanism.
  3. Can the authors extend the evaluation to stronger and larger general-purpose and medical LLMs?
  4. Although the paper includes a simulated non-IID hospital partition, can the authors provide evaluations on real multi-institutional datasets or under more diverse institutional distributions to further validate its practical applicability?

References
[1] RareAgents: Autonomous Multi-disciplinary Team for Rare Disease Diagnosis and Treatment, AAAI, 2026.

2 comments
AuthorsOpenReview ↗

Response to Reviewer RQoj [1/2]

We sincerely thank the reviewer for the thoughtful review and for recognizing that LatentDx enables collaborative diagnosis without directly sharing private clinical records, and that our experiments cover multiple backbones, cross-family collaboration, non-IID hospital partitions, and multiple privacy attacks. We address each concern below. New experiments are marked (new) below and in blue in the revised PDF.

W1 & Q1: Verifiable Diagnostic Evidence

We thank the reviewer for raising this point. LatentDx keeps remote records inside their hospitals by design, so the host does not read them directly. Rather than sharing records in bulk, LatentDx supports verification on demand.

Ranked differential with calibrated confidence (new). Beyond a single disease name, the host can score all 7,454 OMIM disease names after receiving the hospital blocks and return a ranked differential diagnosis (Appendix D.5). On Llama-3.2-3B, the correct disease appears in the top 3 for 88% of test cases and in the top 10 for 93%. After temperature scaling, the expected calibration error falls to 0.04, which matches the error expected from a perfectly calibrated model on 401 cases. The confidence can therefore be used to flag uncertain predictions: answering only the 50% most confident cases yields an accuracy of 0.96, while the remaining cases are referred to the clinician or to expert consultation.

Targeted access to the source record. When a clinician needs to inspect the underlying evidence, the specific record can be requested from the hospital that holds it through standard data-access or consultation procedures. Records thus leave an institution only when needed, one record at a time and under approval, rather than being transmitted for every query.

Clinical validation. Clinically validated explanations require a prospective study with clinicians, which we leave to future work. As stated in our Ethics Statement, LatentDx is intended to assist, rather than replace, clinical judgment, and real use would require clinical validation and expert review.

W2 & Q2: Comparison with Medical Multi-Agent Frameworks

We thank the reviewer for this suggestion.

RareAgents baseline (new). Following it, we re-implemented RareAgents (Chen et al., 2026) and added it to Table 1 under the same benchmark, hospital partition, backbones, and evaluation protocol. In RareAgents, an attending-physician agent convenes a team of specialist agents, conducts rounds of discussion, and makes the final diagnosis. As its agent code is not publicly available, we re-implemented it following the paper, using the host model for every agent. It receives the same evidence as Distributed RAG, namely the top-1 retrieved case from each hospital in plain text.

Despite having direct access to the retrieved records, which exposes private patient information across institutional boundaries, RareAgents remains substantially below LatentDx in both accuracy and macro F1, while LatentDx transmits no clinical text.

AuthorsOpenReview ↗

Response to Reviewer RQoj [2/2]

W2 & Q3: Larger General-Purpose and Medical LLMs

We thank the reviewer for this suggestion. The original submission covers four hosts from three model families and from 3B to 8B parameters, including general-purpose and medical models (Qwen3-4B and Llama-3.2-3B in Table 1, and the medically pretrained MediPhi-3.8B and the larger Llama-3.1-8B in Appendix D.1, Table 16). Following the suggestion, we add two further 8B hosts, the general-purpose Qwen3-8B and the medical UltraMedical-8B (Table 16).

HostTypeBest Baseline Acc / F1LatentDx Acc / F1
Llama-3.1-8BGeneral0.63 / 0.520.70 / 0.63
Qwen3-8B (new)General0.62 / 0.520.70 / 0.63
UltraMedical-8B (new)Medical0.56 / 0.440.74 / 0.67

LatentDx remains the best method on all three 8B hosts, with the largest margin on the medical model.

Q4: Real Multi-Institutional Data

We thank the reviewer for this question. To our knowledge, no large-scale public rare-disease dataset with institution labels is currently available. At the same time, CrossRare-Bench is itself built from real patient cases collected independently by different clinical and research sources. We therefore evaluate two complementary forms of institutional heterogeneity.

Disease-skewed partition. The disease-skewed Dirichlet partition in Appendix D.3 concentrates each disease in one hospital, which holds on average 65% of its cases, and LatentDx remains the strongest method (0.70 vs. 0.42 accuracy for the strongest baseline on Llama).

Source-defined hospitals (new). In response to this question, we further assign each source cohort to its own hospital (Appendix D.4). These independently collected cohorts differ substantially in size and disease coverage. LatentDx is best on every cohort and every metric (0.76 vs. 0.47 overall accuracy on Llama), and its accuracy does not drop relative to the balanced partition (0.73, Table 1).

Reviewer_6G6POpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents LatentDx, a framework for cross-hospital rare-disease diagnosis when hospitals are allowed to share a query but cannot send retrieved clinical records to one another. Each hospital retrieves a relevant case from its local database, encodes that case together with the query, and sends a compact latent representation to a host model. LatentDx-H is designed for hospitals using the same backbone and communicates distilled KV blocks; LatentDx-X handles different model families by projecting compact hidden-state sequences into the host model’s representation space. The authors also build CrossRare-Bench from 8,022 cases spanning 235 OMIM diseases, split the training data across five simulated hospitals, and evaluate both diagnostic utility and information leakage. Across several backbones, LatentDx performs much better than the training-free latent baselines and comes reasonably close to supervised methods that directly share text. Its latent states also reveal less source information than Raw KV and LatentMAS under the attacks considered, though they are clearly not private in any formal sense.

Critical strengths
  1. The problem is worth studying. Cross-hospital inference is different from standard federated learning: the goal is not just to learn from distributed data ahead of time, but to use case-specific evidence stored at another institution for the current patient. Asking what hospitals should communicate in that setting is a good research question.

  2. The method is straightforward and internally consistent. LatentDx-H compresses each local prompt into a fixed-length latent KV block while keeping the underlying LLMs frozen. LatentDx-X adds a simple LayerNorm-plus-linear projector when the sender and host use different model families. There are not many moving parts, and the two variants fit together cleanly.

  3. The evaluation covers a lot of ground. The experiments include several model families and scales, same-backbone and cross-family communication, structured HPO inputs, synthetic clinical narratives, balanced and skewed hospital partitions, and ablations over hospital count, retrieval depth, latent length, and training-set size.

  4. The leakage evaluation is one of the strongest parts of the paper. The authors do not assume that latent communication is automatically safe. They test prompt reconstruction, template-aware extraction, attribute probing, inversion, and membership inference. Raw KV and LatentMAS leak heavily under several of these attacks, which makes the motivation concrete rather than hypothetical.

  5. The MLP-Pool32 comparison is a useful control. It matches LatentDx on diagnosis supervision and latent length, yet performs worse on both utility and leakage. That suggests the result is not explained only by using a short trained bottleneck; how the latent representation is constructed appears to matter.

  6. The supervised text-sharing results are informative. LatentDx gets close to Distributed RAG with prefix tuning or host-side LoRA while avoiding direct transmission of the retrieved case text. On Llama-3.2, for example, LatentDx matches the LoRA baseline’s macro F1 and trails its accuracy by three points.

  7. The systems section is fairly complete. Communication size, latency, memory, host-side re-encoding cost, and one-time training cost are all reported. In the simulated three-hospital Llama setup, LatentDx runs at roughly the same speed as Raw KV while sending a much shorter representation

  8. Reproducibility is taken seriously. The appendices include prompts, hyperparameters, attack architectures, access patterns, and ablations. The paper also provides code and data repository, an AI-use statement, and an appropriate warning that the system is not ready to replace clinical judgment.

Critical weaknesses
  1. The main utility comparison is not fully fair because LatentDx is trained and most of the baselines are not. Table 1 places a diagnosis-supervised interface next to eight training-free methods. The more relevant supervised comparisons—prefix tuning, LoRA, and separately trained hospital-count controls—are pushed into the appendix. This really matters. On Llama-3.2, the trained query-only, one-hospital, and three-hospital systems reach 0.64, 0.69, and 0.73 accuracy. A large part of the apparent gain over the main-table baselines therefore comes from supervision, not necessarily from cross-hospital communication.

  2. It is not yet clear how much the system actually relies on remote retrieved evidence. Every hospital prompt contains both the query phenotypes and a retrieved case. Adding hospitals therefore does three things at once: it introduces more retrieved evidence, repeats the query encoding, and increases the number of latent positions available to the host. The current hospital-count experiment does not separate those effects. The addition of controls where retrieved cases are shuffled, replaced with random or similarity-matched records, stripped of their disease labels, or removed entirely while leaving the repeated query in place. Keep the total latent budget fixed across different values of �0��0�. A counterfactual experiment in which remote evidence is intentionally wrong would be especially revealing.

  3. The benchmark is closed-set and filters out much of the rare-disease tail. Diseases with fewer than ten examples are removed, leaving 235 classes, all of which appear in the training split. The test set therefore measures recognition among diseases already seen during training. It does not test unseen diseases, and it largely avoids the extremely sparse cases that motivate rare-disease diagnosis in the first place. Adding disease-held-out and open-set splits, along with few-shot-by-disease evaluations could help. Top-�1��1� recall, calibration, and abstention would also be more clinically meaningful than top-1 accuracy alone.

  4. The paper needs a cleaner distinction between useful diagnostic information and private record information. The retrieved disease label is treated as a secret, but it is also exactly the kind of signal the communication channel is trained to preserve. Some disease information must survive if the representation is going to help diagnosis. The more interesting question is how much record-specific information survives beyond what the legitimate task requires.

  5. The threat model is still fairly narrow. Most experiments assume a passive observer. The paper does not test repeated observations of the same hospital, chosen or adversarial queries, attacks that combine all hospitals’ blocks, host–hospital collusion, transfer attacks, or substantially larger pretrained decoders. Only two cross-family directions receive the full leakage evaluation.

  6. The data audit and uncertainty analysis could be stronger. Exact duplicates and repeated case identifiers are removed, but the paper does not report semantic near-duplicate checks across sources or attempts to identify the same patient appearing in more than one repository. Because the datasets are public, pretraining exposure is another possible confound. The test set contains 401 examples and 161 represented classes, so macro F1 may also be noisy.

  7. CrossRare-Bench-Text is not a substitute for real clinical notes. The narratives are generated from HPO term names, with disease labels and genes withheld. That is a sensible controlled experiment, but real notes contain negation, uncertainty, irrelevant findings, family history, missing information, and longitudinal context.

  8. The HPO retriever relies on precomputed sentence embeddings and information-content weights, but the exact embedding model, ontology version, IC corpus, and preprocessing are not clearly specified in the main method description. The construction of the balanced partition, the disease-name-to-OMIM mapping, checkpoint revisions, and the exact architecture of the �2��2� control also need more detail. The choice to concatenate local KV blocks while preserving their original rotary phases is unusual and deserves an ablation against re-rotated or globally positioned alternatives.

  9. In the balanced split, each hospital contains between 221 and 226 of the 235 diseases. In other words, nearly every hospital sees nearly every diagnosis class. The Dirichlet experiment is useful, but it is still a synthetic repartition of the same public corpus rather than a test across genuinely different institutions. The authors should treat the source datasets as separate institutions, or create source-held-out experiments that preserve differences in collection and annotation. More realistic tests would vary disease prevalence, phenotype missingness, coding conventions, and ontology coverage across hospitals.

Questions for authors
  1. How much of the reported utility gain comes from diagnosis supervision rather than from LatentDx itself? The main table compares LatentDx with eight training-free baselines, while the closer supervised comparisons appear mostly in the appendix. Could you bring the supervision- and parameter-matched results into the main evaluation and report the gain that remains once training is controlled for?

  2. Can you show that the host is actually using the remote hospitals’ retrieved cases? Each hospital encodes both the query and a retrieved record, and adding hospitals also adds more latent positions. A control with shuffled, random, label-masked, or query-only hospital evidence, while keeping the total latent budget fixed, would help separate the value of remote evidence from repeated query encoding and extra capacity.

  3. How far do you expect these results to generalize beyond the closed-set, simulated hospital benchmark? CrossRare-Bench keeps 235 frequency-filtered diseases and creates hospitals by partitioning public cases. Could you test unseen diseases, source-held-out data, or institution-style shifts in phenotype coverage, coding, and disease prevalence?

  4. What privacy claim are you comfortable making given the remaining leakage? LatentDx is much safer than Raw KV under your attacks, but the inversion decoder still recovers the exact OMIM label in roughly 38–40% of cases, and membership TPR reaches 17% in one setting at 1% FPR. How would the method hold up against repeated queries, joint attacks over all hospital blocks, host collusion, or a larger adaptive decoder?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 6G6P [1/4]

We sincerely thank the reviewer for the thorough and constructive review, and for recognizing the importance of inference-time cross-hospital diagnosis and the leakage evaluation as one of the strongest parts of the paper. The reviewer's concrete suggestions were very helpful, and the revision adds more than ten new experiments, including (1) a main-text comparison with supervised baselines, (2) evidence controls at a fixed latent budget, (3) diseases unseen in training, ranked differentials, calibration and abstention, (4) a stronger attacker, the full cross-family leakage evaluation and a differentially private variant, (5) an ablation of block positions, and (6) source-defined hospitals, a held-out source, and shifts in phenotype completeness and coding granularity. We address each point below. New experiments are marked (new) below and in blue in the revised PDF.

W1 & Q1: Gain Once Training Is Controlled

We thank the reviewer for raising this point, and we agree that supervised and training-free methods should be compared on equal footing.

Supervised comparisons in the main text. As suggested, the revision adds Section 4.4 and Table 3 to the main text, which compare LatentDx with baselines trained on the same diagnosis labels.

Gain from hospital evidence. With training controlled, hospital evidence still adds substantially. The query-only model (N=0N=0) is trained with the same distiller, labels and recipe but without hospital evidence. It reaches 0.64 accuracy against 0.73 for LatentDx on Llama, and 0.41 against 0.64 on Qwen (Appendix E.2, Table 29).

Same latent budget (new). To also match the latent budget, we train a query-only model that sends three blocks of the same size as LatentDx but without retrieved cases. Averaged over three training seeds, LatentDx exceeds it by 0.07 accuracy and is better on all three metrics (Table 30).

Other trained methods. LatentDx outperforms MLP-Pool32, a supervised latent interface with the same latent length, and is competitive with Dist. RAG and Struct. RAG trained with prefix tuning or host LoRA (new for Struct. RAG, Table 3). The RAG baselines send the retrieved records or disease labels to the host as text, whereas LatentDx sends only latent blocks, which greatly reduce leakage (Table 4). LatentDx also keeps the host frozen and trains only a 9.45M-parameter distiller, against 24.3M parameters for host LoRA.

W2 & Q2: Does the Host Use the Remote Evidence?

We thank the reviewer for this suggestion, which directly tests a central assumption of LatentDx.

Evidence controls at a fixed latent budget (new). We keep the main checkpoint and the latent budget fixed, so that every hospital still sends one block, and change only the evidence at every hospital at test time (Appendix E.2, Table 31). Following the reviewer's suggestion, we replace the retrieved case with (1) the case retrieved for another query, (2) a random record, (3) the most similar case of another disease, or (4) the same case with its disease name replaced by a wrong one. Accuracy falls from 0.73 to between 0.10 and 0.30 under every change.

Counterfactual evidence. Settings (3) and (4) are the counterfactual the reviewer proposed. When the evidence points to a wrong disease, the host names that disease in 42% and 47% of the cases, respectively. In contrast, when the substituted evidence does not fit the query, as in (1) and (2), the host rarely follows it (below 10%). The host therefore reads the content of the remote evidence, rather than relying on the repeated query encoding or the extra latent positions.

Query without evidence. The same-budget query-only model in W1 removes the retrieved cases entirely while keeping the query, and LatentDx exceeds it by 0.07 accuracy (Table 30). Together, these controls separate the value of remote evidence from repeated query encoding and latent capacity.

AuthorsOpenReview ↗

Response to Reviewer 6G6P [2/4]

W3 & Q3: Closed-Set Evaluation and the Rare-Disease Tail

We thank the reviewer for these suggestions, which make the evaluation more clinically meaningful.

Diseases unseen in training (new). We built a strict split (Appendix D.6) in which 58 diseases are removed from all training queries and training-time hospital databases, while the test-time databases still contain them. We find this setting genuinely challenging, since the interface never saw these diseases during training. To address it, we add name-anonymized copies of half of the training queries, in which every disease name is consistently replaced by an unrelated one so that the answer must be read from the evidence rather than recalled. With this change, LatentDx reaches 0.47 accuracy on unseen diseases, nearly double its original accuracy, while accuracy on seen diseases remains high (Table 23).

The rarest diseases (new). The threshold of ten cases per disease is needed to split each disease across hospitals and to evaluate it reliably, and diseases below this threshold are closest to the unseen-disease setting above. Within the benchmark, we further group the test cases by the number of training cases of their disease. LatentDx is the most accurate method in every group, including the rarest diseases, which keep only 8 or 9 cases for training (0.67 against at most 0.38), and those with 10 to 19 training cases (0.60 against at most 0.40, Appendix G, Table 40).

Ranked differential, calibration and abstention (new). Beyond a single disease name, the host can score all 7,454 OMIM disease names, rather than only the 235 benchmark diseases, and return a ranked differential diagnosis (Appendix D.5). On Llama, the correct disease appears in the top 10 for 93% of test cases. After temperature scaling, the expected calibration error falls to 0.04, and answering only the 50% most confident cases yields an accuracy of 0.96, while the remaining cases are referred to clinicians.

Institutional shifts. Source-held-out data and institution-style shifts in phenotype coverage, coding and disease prevalence are addressed in our response to W9.

W4: Task Information versus Record Information

We thank the reviewer for this insightful point, and we agree with the distinction. Some disease information must survive for the remote evidence to be useful, so the retrieved disease label is in part task information. The more meaningful question is how much record-specific information survives beyond it.

Disease-level information. The inversion decoder recovers the retrieved disease from LatentDx blocks in 38% of cases, against 27% without any transmission and nearly all cases for Raw KV (Table 4). This moderate excess over the control is consistent with the disease signal that the diagnostic task requires.

Record-specific information. Beyond the disease, the record-specific content is the retrieved patient's phenotype profile and clinical text. For the inversion decoder, the recovery of the retrieved record's phenotype terms stays close to the no-transmission control (HPO micro-F1 0.27 vs. 0.23 on Llama), whereas Raw KV reaches 0.87 (Table 37). Prompt reconstruction likewise recovers no more of the source text than the control (token F1 0.07 for both on Llama, Table 4).

Not merely a weaker channel. MLP-Pool32, trained with the same supervision and latent length, achieves lower utility yet exposes more disease information to the probe and the inversion decoder (Appendix E.1). LatentDx therefore does not reduce leakage simply by transmitting less, and how the block is constructed determines how much it carries beyond the task. We have made this distinction explicit in Section 4.6.

AuthorsOpenReview ↗

Response to Reviewer 6G6P [3/4]

W5 & Q4: Threat Model and Privacy Claim

We thank the reviewer for this question, which helped us state the scope of our privacy claim precisely.

The claim. We claim that, under the evaluated attacks, LatentDx substantially reduces the record content recoverable from the transmitted messages compared with Raw KV and LatentMAS, and that LatentDx-DP also protects membership against an attacker who holds the complete record. We have also revised the threat model to state that the attacker may know all weights, including the trained interface (Section 3.1).

Chosen queries and membership (new). Our membership attack already lets the attacker choose the query, using the candidate's own phenotypes. Following Reviewer 5ymK, we also evaluate a stronger attacker who holds all weights and the complete candidate record and recomputes the deterministic block. This attacker detects membership (TPR 1.00 at 1% FPR), as it does for Raw KV and LatentMAS, but its power relies on the complete record, and with 90% of the candidate's phenotypes its TPR falls to 0.43. To protect membership against this attacker, we developed LatentDx-DP, in which each hospital encodes a differentially private summary of its local evidence. Each query satisfies (4.1,10−5)(4.1, 10^{-5})-DP, which lowers this attack to 0.02 at a cost of 0.01 accuracy (Appendix F.5).

Repeated queries. Block construction in LatentDx is deterministic, so repeating the same query returns the identical block and reveals nothing new. Under LatentDx-DP, the per-query guarantee composes over the queries a hospital answers, which the hospital can bound with a query budget.

Stronger attackers (new). An inversion decoder about three times larger (140M parameters) recovers the retrieved disease in 37% of cases, against 27% without transmission, the same level as the original decoder (Appendix F.3). All learned attackers are trained directly on blocks produced by the deployed distiller, which makes them at least as strong as transfer attacks that must instead train on blocks from a surrogate model.

All cross-family directions (new). Every cross-family direction of Table 2 now has the full five-attack evaluation (Appendix F.4, Table 38), and leakage stays far below Raw KV and LatentMAS in every direction.

Joint attacks and collusion. Each hospital computes its block only from the shared query and its own retrieved record, and the blocks do not interact before they reach the host. Beyond what a hospital's own block reveals, a joint attack over all blocks can therefore learn about its record only through what the other hospitals' records have in common with it. For membership, the compute-and-compare attacker above already represents the worst case, since it holds all weights and the complete record. Under LatentDx-DP, combining blocks from several hospitals, or collusion between the host and other hospitals, is post-processing of the released summaries and cannot weaken the per-query guarantee for any hospital's records.

W6: Data Audit and Uncertainty

We thank the reviewer for these suggestions.

Near-duplicates across sources (new). Appendix A.1 now reports a near-duplicate audit, which also covers the same patient appearing in more than one repository. No test case shares its HPO set with a training case of the same disease, and 11 of the 401 test cases (2.7%) have one with HPO Jaccard similarity of at least 0.9. Removing these 11 cases changes the accuracy of every method by at most one point. Likewise, when hospitals never return a case with Jaccard of at least 0.9 (or 0.8) to the query, the accuracy of LatentDx changes by less than one point.

Pretraining exposure. Without retrieved evidence, the zero-shot baseline reaches only 0.01 accuracy on Llama and 0.03 on Qwen (Table 1). The backbones therefore cannot diagnose these cases from the phenotypes alone, which argues against memorized case-label pairs from pretraining.

Uncertainty (new). On Llama, a paired bootstrap over the 401 test cases gives 95% confidence intervals for the accuracy and macro F1 gains of LatentDx over every baseline in Table 1 that all exclude zero (Appendix G, Table 40). Across three training seeds, accuracy is 0.73±0.010.73 \pm 0.01 on Llama (Appendix D.7).

AuthorsOpenReview ↗

Response to Reviewer 6G6P [4/4]

W7: Synthetic Clinical Narratives

We thank the reviewer for this point, and we agree that CrossRare-Bench-Text is not a substitute for real clinical notes. It is a feasibility study, which tests whether LatentDx can carry evidence written as free text rather than as structured HPO terms. In this setting, LatentDx remains the most accurate method overall on all three backbones (Appendix D.2, Table 17). Real clinical notes of rare-disease patients are difficult to obtain and to share across institutions, which is the very constraint that motivates LatentDx, so we leave evaluation on real notes to future work and now state this purpose in Appendix D.2.

W8: Implementation Details and Block Positions

We thank the reviewer for the careful reading.

Implementation details. The revision adds Appendix B.4, which specifies the retrieval embedding model, the HPO release and the information-content weights, the partition procedure, the mapping from predicted names to OMIM, the N=0N=0 control, and the commit of every backbone. In particular, the information-content weights are computed from disease annotations in the RareBench knowledge graph, so no benchmark case enters them.

Block positions (new). Each block keeps the rotary phases of its own local prompt, so the host receives exactly what the hospital computed. As suggested, we also train and evaluate a variant in which every block is re-rotated to its global position in the host sequence. It reaches similar accuracy to local phases (Appendix E.4). We keep local phases because a block then does not depend on its slot in the host sequence, so the host can concatenate blocks from any number of hospitals in any order without re-rotating them.

W9: Realism of the Hospital Partition

We thank the reviewer for this suggestion. We have added several experiments that model differences across institutions, as described below.

Source-defined hospitals (new). In the revision each source cohort becomes its own hospital (Appendix D.4). The three hospitals differ in size (5,894 / 804 / 522 cases), disease coverage (219 / 38 / 64 diseases) and annotation practice, so two of the three hospitals hold only a small part of the 235 diseases. LatentDx is best on every cohort and every metric, with 0.76 overall accuracy against 0.47 for the strongest baseline.

A source held out from training (new). We also retrain the distiller without any Zenodo record, removing them from the training queries, the validation queries and the training-time hospital databases, so that Zenodo joins as its own hospital only at test time. LatentDx still has the highest accuracy of all methods on the Zenodo patients (0.55 vs. 0.41 for the strongest baseline, Appendix D.4, Table 21).

Disease prevalence. The disease-skewed partition in Appendix D.3 concentrates each disease in one hospital, which holds on average 65% of its cases, so the prevalence of a disease differs sharply across hospitals. LatentDx keeps the highest accuracy there on both backbones.

Phenotype missingness (new). To simulate hospitals that record phenotypes less completely, each test query keeps only a random 80% or 60% of its HPO terms. LatentDx still reaches 0.67 accuracy with 60% of the terms, against at most 0.43 for the strongest baselines (Appendix D.4).

Coding conventions (new). To simulate a hospital that codes at a coarser level, we replace half of the HPO terms of each test query by their most specific parent term. LatentDx reaches 0.57 accuracy, against at most 0.38 for the strongest baselines (Appendix D.4).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and New Experiments

We thank all reviewers for their constructive feedback. Following their suggestions, we added more than fifteen new experiments and analyses, listed below. All changes are marked in blue in the revised PDF and discussed in the individual responses.

Privacy

  • Compute-and-compare membership attack with the complete candidate record (Appendix F.5)
  • LatentDx-DP, a variant with a per-query differential privacy guarantee (Appendix F.5)
  • A larger inversion decoder (Appendix F.3)
  • The full five-attack evaluation in every cross-family direction (Appendix F.4)

Realism and generalization

  • Hospitals defined by source cohort, and a source held out from training (Appendix D.4)
  • Shifts in phenotype completeness and coding granularity (Appendix D.4)
  • Diseases unseen in training (Appendix D.6)
  • A near-duplicate audit across sources (Appendix A.1)

Utility and baselines

  • Comparison with supervised baselines in the main text (Section 4.4)
  • Evidence controls and a query-only control at a fixed latent budget (Appendix E.2)
  • Struct. RAG trained with prefix tuning and host LoRA (Appendix D.9)
  • RareAgents as a medical multi-agent baseline (Table 1)
  • Two larger models, Qwen3-8B and UltraMedical-8B (Table 16)
  • Ranked differential diagnosis, calibration and abstention (Appendix D.5)
  • Local versus re-rotated block positions (Appendix E.4)
  • Paired bootstrap confidence intervals and accuracy by disease frequency (Appendix G)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.