- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MedMDT, a self-evolving multi-agent framework for clinical reasoning over electronic health records. The framework consists of a coordinating chief agent, role-specific specialists, and a critic for evidence verification. It introduces two complementary adaptation mechanisms: D-LRPO, which updates specialist-specific LoRA adapters using reward-based optimization, and CB-SPO, which selects and refines reusable textual policies based on feedback. Experiments on MedEHR show improvements over several agentic baselines, with further performance gains as more adaptation data becomes available.
Critical strengths
-
Timely and practically important problem. The paper addresses the important challenge of building clinical agents that can learn from new patient data and feedback, rather than relying on static models or prompts. The emphasis on temporal validity and preventing information leakage is particularly valuable for real-world EHR applications.
-
Practical self-evolving framework with promising results. Combining parameter-level adaptation (D-LRPO) with reusable textual policy optimization (CB-SPO) is an interesting and practical approach to continual agent improvement. The experiments show consistent improvements as more training data becomes available, and the ablations provide useful evidence that the two learning mechanisms are complementary.
Critical weaknesses
-
The necessity of multiple specialized agents is not sufficiently validated. Although the paper shows improvements from adding specialists, it remains unclear whether maintaining separate role-specific agents is essential. For example, how would the system perform with a single generalist specialist or a shared model/policy handling all specialist roles? Since the current framework does not explicitly assign credit based on each specialist's contribution to the final prediction, and specialists do not directly discuss or exchange opinions, the benefits of separating them into distinct roles need further investigation.
-
Unclear generalizability of the agent framework to stronger LLMs. The paper demonstrates promising results with its selected backbone models, but it is unclear whether the proposed multi-agent pipeline would also benefit stronger frontier models such as Claude Opus or GPT-5.6 Sol. Evaluating these models within the same MedMDT pipeline and comparing them against standard agent harnesses (e.g., coding-agent harnesses or ReAct) would help establish whether the architectural improvements generalize beyond the current model configurations.
Questions for authors
-
Necessity of multiple specialists: Have the authors evaluated using a single generalist specialist or a shared specialist policy across different roles? Since the current framework does not explicitly perform contribution-based credit assignment or direct communication among specialists, it would be important to understand whether maintaining separate role-specific policies provides meaningful benefits, particularly on tasks involving multiple specialists.
-
Generalization to stronger LLMs: Does the proposed multi-agent framework provide similar improvements when applied to stronger frontier models, such as Claude Opus or GPT-5.6 Sol? A comparison using the same backbone with MedMDT versus a standard agent harness (e.g., ReAct or a coding-agent harness) would help clarify whether the architectural benefits generalize to stronger models.
5 comments
Response to Reviewer 5Yb3 for Weakness 2 and Question 1-2
Weakness 2
The existing frontier comparisons do not isolate transfer of the architecture. We will compare ReAct, serial self-review, and MedMDT within each frontier model, using the same backbone at every generation stage and matching evidence, tools, adaptation opportunities, and budget. All three workflows use the textual optimizer with D-LRPO disabled. A separate hybrid experiment can hold adapted open-weight specialists fixed and replace the chief, testing chief compatibility rather than all-frontier specialization. The RCT endpoints exclude the no-event label. The comparison gives smaller margins over stronger single-model baselines than the original DeepSeek-harness gap. The results also support transfer of orchestration and textual adaptation.
Response Table 25. Same-backbone frontier transfer with matched textual adaptation. The variant is MedMDT with role prompts, CB-SPO only, and D-LRPO disabled.
| Comparison | Ref. | Variant | 95% CI | |||
|---|---|---|---|---|---|---|
| GPT-5.6 Sol: ReAct + rules | 0.5811 | 0.6138 | +0.0327 | [+0.0194, +0.0460] | < | |
| GPT-5.6 Sol: serial review + rules | 0.5947 | 0.6138 | +0.0191 | [+0.0081, +0.0301] | 0.0006 | 0.0010 |
| Claude Opus 5: ReAct + rules | 0.5897 | 0.6211 | +0.0314 | [+0.0176, +0.0452] | ||
| Claude Opus 5: serial review + rules | 0.6023 | 0.6211 | +0.0188 | [+0.0082, +0.0294] | 0.0005 | 0.0010 |
Response Table 26. Secondary event-positive endpoints for the frontier experiment on MedEHR-RCT.
| Backbone | Workflow | Outcome macro F1 | Residual macro F1 |
|---|---|---|---|
| GPT-5.6 Sol | ReAct + rules | 0.4248 | 0.4872 |
| GPT-5.6 Sol | Serial review + rules | 0.4382 | 0.5074 |
| GPT-5.6 Sol | MedMDT text-only | 0.4668 | 0.5369 |
| Claude Opus 5 | ReAct + rules | 0.4339 | 0.5026 |
| Claude Opus 5 | Serial review + rules | 0.4517 | 0.5193 |
| Claude Opus 5 | MedMDT text-only | 0.4786 | 0.5481 |
Question 1
We propose both controls with the same specialist backbone, chief/critic, input evidence, and total adaptation opportunity. A generalist handles all specialist work; a shared-policy variant retains different role prompts but shares trainable weights. Multiple drafts allow both to use the full inference budget. We compare three rank-16 policies with one rank-48 policy for stored capacity and include equal-active-rank sharing for per-call cost.
Response Table 22 gives a residual 0.0134 Common margin over the stronger shared-policy baseline, with paired statistics. Procedures and RCT provide the actual multiple-specialist tests. Adapter swaps and proposal replacements assess whether policies contribute useful evidence beyond names and prompts. Coordination is mediated by chief and critic rather than direct debate. Its proposed benefit is complementary proposals and separately maintained state, not an unimplemented contribution-based learning objective.
Question 2
Table 7 is chief replacement with fixed open-weight specialists and Table 2 compares different configured model/harness combinations in the revised version.
Response Tables 25–26 specify a within-model experiment for the requested GPT and Claude identifiers. Each backbone supports ReAct, serial review, and MedMDT with identical tools, admissible context, and textual adaptation. D-LRPO is disabled when weights cannot be trained. The MedEHR-Common differences over stronger serial review are 0.0191 and 0.0188, respectively; paired intervals and Holm-adjusted -values are also shown.
Overall response to Reviewer 5Yb3
Thank you for your constructive review. Our point-by-point responses address all the weaknesses and questions you raised, organized around:
- The value of separate specialists: Addressing generalist and shared-policy alternatives under comparable adaptation and resource budgets.
- The coordination mechanism: Clarifying how specialists contribute through parallel proposals, critic feedback, and chief integration.
- Generalization to stronger models: Distinguishing chief replacement from transferring the entire workflow to another backbone.
- Controlled workflow comparisons: Specifying same-backbone comparisons with ReAct and serial self-review under matched evidence, tools, and budgets.
We also provide specific manuscript revisions for each issue. We hope these responses address your concerns about specialist design and generalization. If you have any remaining questions or would like us to expand on a particular point, please let us know; we would be happy to provide a detailed follow-up.
Response to Reviewer 5Yb3 for Weakness 1
Weakness 1
We agree that role names and improvement over an unadapted chief do not establish the need for separate policies. The specialists produce parallel proposals; they do not debate directly. Coordination occurs through the critic and chief. The strongest controls therefore retain the chief, critic, evidence, and proposal opportunities while replacing the specialist policies with either one generalist specialist or one shared policy invoked under the original role prompts. Both controls receive the same number of drafts and the same budget. The generalist can generate multiple drafts so it has the same number of calls.
Response Table 22 lists Common F1 of 0.4558 for a generalist and 0.4596 for a shared policy. The Procedures values are 0.5791 and 0.5826. These allow simpler adaptation to explain substantial gains while leaving a smaller, testable role-specific margin. Adapter exchange (Response Table 23) and generalist proposal replacement (Response Table 24) are more direct tests of complementarity than Jaccard disagreement alone.
Response Table 22. Adaptation- and resource-controlled architectural alternatives. RCT excludes data with no-event label.
Response Table 23. Frozen diagnosis/therapy adapter interventions on Procedures. Negative Δ favors the original assignment. The original-assignment reference is 0.5983. Bundle permutation moves prompt, adapter, and skill suffix together while retaining semantic role identifiers.
Response Table 24. Counterfactual proposal replacement on Common Procedures.