- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The authors introduce a framework for correcting the mistakes produced by agents when issuing clinical care plans. Instead of relying on a fixed checklist or written summary they use Wilson’s bounded prevalence mining to learn the required actions from other patients in the cohort and then issue atomic edits to the agents plans. The authors evaluate their approach on a single dataset with DeepSeek V4 Flash and show that their methods provides gains in the satisfaction of hard constraints over popular other LLM based methods such as REACT or Self Reflexion.
Critical strengths
Agents operating within electronic health records is a timely topic and it is important to move beyond the evaluation of endpoint to analyze the trajectories for clinical safety and adequate resource usage.
Critical weaknesses
Experimental evaluation is very weak, only a single model on a small subset of MIMIC patients. Please add evaluation on more datasets such as VivaBench (Chiu et al.), MIMIC Gastro (Hager et al.) and AgentClinic (Schmidtgall et al.) as well results using a variety of SOTA open and closed source models.
Technical novelty is very incremental as it is just a simple verification process for LLM plans using established technologies and there are also no theoretical results.
Questions for authors
I do not agree with the authors statement that free-text plans make process difficult to verify as this is already standard in clinical practice when a pharmacist checks a prescription from a physician and this is unlikely to be replaced by autonomous AI agents in the future.
Furthermore the authors state that hand authored checklists require disease specific engineering but most hospitals already operate under national or local guideline that cover precisely these cases. As such it appears to me that the current systems just learns the correct policy (potentially with noise) from a set of previous patients instead of applying it directly.
Can the authors please add a baseline where the agent is given access to the guidelines (checklist or free text) and asked to ensure all items on there are followed.
Furthermore, could the others please elaborate on the significance of the corrections issues by their system as the LLM outputs of the dataset used were analyzed extensively by Ferber et al. who found it to be safe and guideline concordant.
Why can the reflection and LLM critic baselines not get access to the full history?
Figure does not add much information beyond Table 1 I fele. Space should be used for additional datasets.
Can the authors please run the agent as implemented by Ferber et al. in the original paper since they are using their dataset.
The claim that appendectomies where introduce in PE trajectories donor cohort specifications needs more explanations. Currently it reads like the implemented baseline is degenerate.
There is no clear discussion of the limitation of the approach including the limited evaluation, reliance on prior samples being available (will not be the case for rare diseases) and the problem of adapting to shifts in the guidelines as historical patients will have been treated under the old guidelines.
Minor comments: Missing space before Ferber et al on page 1 Figure 1 is very cluttered Table 1 caption please state what is plotted to make it easier to parse