- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper focuses on the traceability of the diagnostic process for multi-agent rare disease reasoning systems, aiming to reduce the memory gap across multiple agents. It introduces a memory structure that can be shared by different specialized agents during the diagnosis process, supporting persistent diagnostic state, event tracking, semantic memory, and reasoning. It also designs a complete workflow to integrate the shared memory into the multi-agent diagnostic process.
Critical strengths
-
The paper is well organized. The figures clearly present the overview of the whole framework. The authors state the role of each specialized agent, helping readers to understand the architecture of the proposed framework. In addition, the motivation of this work is reasonable and practical. The authors focus on the traceability of the diagnostic process for the multi-agent disease reasoning systems, aiming to reduce the memory gap among multiple agents.
-
The experiments cover four large language models and two datasets. The proposed framework achieves the highest R@1 on both MIMIC-IV-Note and RareArena datasets. In addition, the analysis of multi-category evaluation and agent usage also provides more information to show the framework's performance.
Critical weaknesses
- The current version of the manuscript lacks some important methodological details that may affect reproducibility. Here are some examples:
- This work defines the agent output in Section 3.1, but does not describe its detailed structure, such as the attributes of this output and the role of each attribute, and whether all the agents with different roles share the same output structure.
- It is not clear how the agent output is used in the subsequent process. For example, Section 3.1 defines the operators and and their high-level use but does not show how this process is performed and how the agent output is merged into the updated and .
- In Section 3.1, is defined for 'retrieving relevant information and constructing the context required by the target agent'. However, the authors do not show the detailed selection rules and construction method.
- In line 7 of Algorithm 1, and are evaluated to decide whether the current evidence is sufficient. However, this work does not define the criteria for 'sufficient evidence'. This work does not show the concrete decision rules or prompt used in the readiness judgement module. In addition, it is important to establish that this judgment is clinically valid and well justified.
- The experiments lack an ablation study and some necessary justification of the experimental settings. This work provides a performance comparison between the static workflow and DiagMem. However, it does not include an ablation study that isolates each core module to demonstrate the effectiveness and necessity of the proposed modules. Compared to the static workflow, DiagMem changes both the memory structure and agent workflow. The lack of an ablation study makes the contribution of each module unclear. In addition, Section 4.1 states that 'we filtered cases with only one disease outcome and randomly selected 1,000 cases for evaluation'. It would be better for the authors to provide a more detailed justification for selecting only cases with only one disease outcome and whether this indicates any limitation of the current framework.
Questions for authors
- To ensure reproducibility, could the authors provide the necessary implementation details and prompts listed in the first point of Weaknesses?
- Could the authors give a more detailed justification for selecting cases with only one disease outcome and clarify whether this selection reflects any limitations in the capabilities of the current framework?