acceptodds
Withdrawn from ICLR 2027

DiagMem: Closing the Diagnostic Memory Gap for Evidence-Grounded Multi-Agent Rare Disease Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Rare disease diagnosis often depends on evidence scattered across symptoms, laboratory results, family history, phenotypes and external disease knowledge. Current LLM-based diagnostic systems usually treat this process as prompt-level prediction. They may produce plausible disease rankings, but they do not maintain what has been established, which hypotheses have been considered, what remains uncertain or how new evidence changes the case. We introduce Patient Diagnostic Memory, a shared-memory framework for multi-agent rare disease diagnosis. For each patient, the framework maintains a diagnostic state, event records and semantic memory entries that store reusable clinical evidence. Specialized agents extract clinical facts, normalize phenotypes with Human Phenotype Ontology terms, interpret laboratory findings, analyze family history and retrieve external disease knowledge. A diagnostic controller reads the evolving memory to decide which evidence source to use next and when to produce a ranked diagnosis. We evaluate the framework on MIMIC-IV-Note and RareArena with four backbone LLMs. Memory-augmented diagnosis improves top-ranked accuracy across all models, with larger gains on rare disease cases. Average top-1 accuracy increases by 14.5 percentage points on RareArena and 4.3 points on MIMIC-IV-Note. These results suggest that patient-level diagnostic memory can make multi-agent clinical reasoning more traceable, modular and evidence-grounded.

Primary area:
foundation or frontier models, including LLMs
Keywords:
rare disease diagnosis, multi-agent LLM systems, shared memory middleware, evidence-grounded clinical reasoning, diagnostic decision support

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_FzwkOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on the traceability of the diagnostic process for multi-agent rare disease reasoning systems, aiming to reduce the memory gap across multiple agents. It introduces a memory structure that can be shared by different specialized agents during the diagnosis process, supporting persistent diagnostic state, event tracking, semantic memory, and reasoning. It also designs a complete workflow to integrate the shared memory into the multi-agent diagnostic process.

Critical strengths
  1. The paper is well organized. The figures clearly present the overview of the whole framework. The authors state the role of each specialized agent, helping readers to understand the architecture of the proposed framework. In addition, the motivation of this work is reasonable and practical. The authors focus on the traceability of the diagnostic process for the multi-agent disease reasoning systems, aiming to reduce the memory gap among multiple agents.

  2. The experiments cover four large language models and two datasets. The proposed framework achieves the highest R@1 on both MIMIC-IV-Note and RareArena datasets. In addition, the analysis of multi-category evaluation and agent usage also provides more information to show the framework's performance.

Critical weaknesses
  1. The current version of the manuscript lacks some important methodological details that may affect reproducibility. Here are some examples:
  • This work defines the agent output Δt(i)\Delta_t^{(i)} in Section 3.1, but does not describe its detailed structure, such as the attributes of this output and the role of each attribute, and whether all the agents with different roles share the same output structure.
  • It is not clear how the agent output is used in the subsequent process. For example, Section 3.1 defines the operators UsU_s and UrU_r and their high-level use but does not show how this process is performed and how the agent output Δt(i)\Delta_t^{(i)} is merged into the updated St+1S_{t+1} and Rt+1R_{t+1}.
  • In Section 3.1, Φ\Phi is defined for 'retrieving relevant information and constructing the context required by the target agent'. However, the authors do not show the detailed selection rules and construction method.
  • In line 7 of Algorithm 1, StS_t and RtR_t are evaluated to decide whether the current evidence is sufficient. However, this work does not define the criteria for 'sufficient evidence'. This work does not show the concrete decision rules or prompt used in the readiness judgement module. In addition, it is important to establish that this judgment is clinically valid and well justified.
  1. The experiments lack an ablation study and some necessary justification of the experimental settings. This work provides a performance comparison between the static workflow and DiagMem. However, it does not include an ablation study that isolates each core module to demonstrate the effectiveness and necessity of the proposed modules. Compared to the static workflow, DiagMem changes both the memory structure and agent workflow. The lack of an ablation study makes the contribution of each module unclear. In addition, Section 4.1 states that 'we filtered cases with only one disease outcome and randomly selected 1,000 cases for evaluation'. It would be better for the authors to provide a more detailed justification for selecting only cases with only one disease outcome and whether this indicates any limitation of the current framework.
Questions for authors
  1. To ensure reproducibility, could the authors provide the necessary implementation details and prompts listed in the first point of Weaknesses?
  2. Could the authors give a more detailed justification for selecting cases with only one disease outcome and clarify whether this selection reflects any limitations in the capabilities of the current framework?
Reviewer_BBJeOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DiagMem, a shared-memory framework for multi-agent rare disease diagnosis, built on the idea that diagnosis should maintain an explicit patient-level record of established facts, open hypotheses and uncertainty rather than map a record to a ranked list in one pass. The authors formulate this as a per-patient diagnostic state, an event log of agent operations and a store of structured clinical findings, and augment a fixed multi-agent workflow with a middleware that updates these stores after each call and a controller that chooses the next agent and decides when to stop. Evaluation uses 1,000 cases each from MIMIC-IV-Note and RareArena with four backbone LLMs and R@1/3/5, against direct prompting and a static workflow without the middleware. Compared with direct prompting, the reported advantages are most pronounced on RareArena and small on MIMIC-IV-Note; compared with the static workflow, the additional advantage is smaller on both datasets.

Critical strengths

S1. The paper addresses a relevant problem: maintaining coherent, traceable evidence across multiple diagnostic agents.

S2. Evaluation across two datasets and four backbones, including a static multi-agent baseline, provides useful initial evidence of the framework's potential.

Critical weaknesses

W1. Inconsistent results undermine reliability. For LLaMA with DiagMem, weighting Table 3 by the reported category sizes yields an overall R@1 of approximately 36.49%, whereas Table 2 reports 45.7%; moreover, every category's R@1 is at most 40.1%, making the reported aggregate impossible under the stated partition. The problem is not confined to LLaMA: the LLaMA "+System" row weights to 0.365/0.595/0.721 — exactly Table 2's Direct row for LLaMA — while its baseline row weights to 0.285/0.386/0.407, which appears nowhere in Table 2; DeepSeek-v4 "+System" weights to 0.648/0.778/0.823 vs. 0.689/0.838/0.873 in Table 2; and 64 of the 120 entries in Table 3 cannot be expressed as k/n for the stated n of their category (e.g., 0.328 with n = 224 lies between 73/224 and 74/224). It is also never stated whether "+System" denotes DiagMem or the static workflow.

W2. Novelty is insufficiently established. The distinction from existing memory-equipped diagnostic systems, particularly DeepRare [1] (central host with long-term memory and traceable reasoning, itself evaluated on MIMIC-IV- and RareArena-derived cases), is neither explained nor experimentally demonstrated; MACD and MedAgents, a RAG-only baseline with the same MSD retrieval, and a CoT direct baseline are also absent.

[1] Weike Zhao et al., "An agentic system for rare disease diagnosis with traceable reasoning", Nature (2026).

Questions for authors

Please see the Critical Weaknesses section.

Reviewer_Pu5dOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DiagMem, a shared-memory multi-agent framework for rare disease diagnosis. It consists of patient-level diagnostic states, event records, and semantic memories to support evidence integration and dynamic agent coordination. Experiments on MIMIC-IV-Note and RareArena show improvements in diagnostic ranking accuracy across four backbone LLMs.

Critical strengths
  1. The framework provides a structured approach to organizing clinical evidence and coordinating specialized agents through shared patient-level memory.
  2. Experiments across multiple LLMs and datasets demonstrate improvements in diagnostic ranking accuracy, particularly for rare diseases.
Critical weaknesses
  1. Limited methodological novelty. Prior work, particularly the closely related RareAgents [1], has already explored memory-augmented multi-agent frameworks for rare disease diagnosis. The paper does not clearly distinguish its contributions from existing approaches. Moreover, the experiments only compare against direct diagnosis and a self-defined static workflow, without competitive multi-agent baselines, making it difficult to assess the actual advantages of DiagMem.
  2. Insufficient implementation details. The paper lacks essential details about agent prompts, memory update strategies, and controller decision criteria. For example, how the Laboratory Analysis Agent identifies clinically meaningful patterns remains unclear, limiting reproducibility.

References
[1] RareAgents: Autonomous Multi-disciplinary Team for Rare Disease Diagnosis and Treatment, AAAI, 2026.

Questions for authors
  1. What are the fundamental differences between DiagMem and existing memory-augmented frameworks such as RareAgents [1]?
  2. Can the authors provide the complete agent prompts and details of the memory update and controller mechanisms?
  3. Although patient-level diagnostic memory is the core contribution, its clinical validity has not been evaluated. Can the authors provide expert assessments of the correctness and clinical relevance of the generated memories, particularly the laboratory findings and diagnostic evidence?
  4. The paper frequently characterizes existing approaches as "prompt-level prediction". Could the authors clarify this definition?

References
[1] RareAgents: Autonomous Multi-disciplinary Team for Rare Disease Diagnosis and Treatment, AAAI, 2026.

Reviewer_V4bMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

the paper has a plausible engineering contribution in rare disease diagnostics with LLMs, it clearly defines what a useful patient state is in the context of medical AI, and proposes agents with complementary abilities such as lab understanding and external knowledge retrieval, that shares memory and gets regularized through a controller.

Critical strengths

The question studied is of real clinical value.

The quantitative ablations and experiments are detailed and clear.

Critical weaknesses

Disclaimer: I have experience in this clinical domain, but i have only read about, not personally used or implemented these methods or related methods in my own work. Also, the AC assigned me as emergency reviewer, and I have other important work to do, so I didn't not read the appendix at all. Dear authors, please indulge me if I make mistakes in the review and point it out as soon as possible.

  1. Can you please show more concretely that whether diagnostic reasoning faithfully follows the clinical evidence? I would like to see in-depth analysis of how contradictions (if any) are handled, cited source actually supports the diagnosis etc.

  2. Lack of implementation details, which is understandable given the short page limit of iclr compared to a journal. but please define clearly in rebuttal, the exact rules or procedures for state updates, extraction, controller, etc. im not saying that the formulas are wrong or poorly defined, its just that text will be more clear. For example, when one agent contradicts another, does the system preserve both claims, prefer the latest, use source reliability, ask another model, or invoke additional tools? The diagnostic state includes a conflict variable, but having a field for conflicts does not specify a conflict-resolution mechanism.

  3. Soundness wise, as someone who has lived with the mimic dataset, I really have to criticize your choice of handling this dataset. i) It's not a good choice for "rare diseases" honestly, in the first place. Good rare disease resources could be PMC papers and case reports, because if something is published as a paper, there may be something unique about the case. MIMIC is more general public population. ii) The filtering is just too random, we seem to filter by "one outcome" and just picks 1k cases? MIMIC -based medical AI papers are sometimes hard to trust if you use a subset (i totally understand you not using full dataset because you are writing an LLM paper which are computationally expensive.) iii) can you answer in your response to me a clearer definition of the data wrangling process? MIMIC has many worksheets and some of them may just unintentionally leak the diagnosis outcome.

  1. I found it a bit hard to connect the introduction's motivation of revising an existing diagnostic state when new clinical evidence arrives to your actual tested use cases, wouldn't that be in the case of sequential decision making? (e.g. computationally, you can design a sequential evidence-release experiment, or so called "multi-turn diagnosis".)
Questions for authors

I stated four confusions or weaknesses above. In your rebuttal, I highly recommend doing a point to point response to help me understand your work better and help me make a fair judgement.

specifically,

a) in-depth analysis of whether diagnostic reasoning faithfully follows the clinical evidence

b) implementation details

c) data soundness

d) explanation of motivation - experiment

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.