- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposed CARing, a LLM and GRPO-based approach for EHR Diagnosis Prediction. In particular, the method first learns a medical concept tokenizer with RQ-VAE and is injected with ICD ontology for enhanced medical knowledge, which is dubbed Semantic IDs (SIDs). In order for the LLM to be able to comprehend and reason on these SIDs, the paper introduced six multi-task alignment tasks between SIDs and clinical text, and to enhance CARing’s reasoning and world knowledge the models utilized knowledge distillation with a strong LLM's reasoning paths. Afterwards, the paper pointed out the mismatch where in standard GRPO rewards each trajectory independently and cannot distinguish repeated correct predictions from coverage of distinct diagnoses. The authors addressed this mismatch with rewards at the visit level, with a coverage reward to avoid collapsing the predictions into a few popular predictions only, and a Multi-positive Supervision SFT signal. The method achieves competitive Diagnosis results on two popular EHR datasets MIMIC-III and MIMIC-IV.
Critical strengths
-
Technically sound framework with clear motivations: The motivations behind each component are clean and logical, which are 1. LLMs can't reason directly on Semantic IDs 2. RL rewards are not fit for unordered multilabel diagnosis, which can cause the model to collapse on a few popular predictions only. The proposed solutions to these problems are technically sound, and proven to work empirically.
-
Competitive experimental results: The method is tested on two popular EHR datasets MIMIC-III and MIMIC-IV, which achieves superior results compared with previous methods.
Critical weaknesses
-
Limited methodological novelty and great similarities to previous works: While the paper clearly positioned their motivations, most of the solutions proposed by the authors are adapted from previous established works. In particular, the idea of injecting ICD codes' representation with its ontology is well-established [1], [2], [3] as well as using codebooks for medical tokenizers [4],[5]. The Multi-task Alignment training tasks between SIDs and text are quite similar to those proposed by existing recommendation papers [6], [7], [8] (using semantic IDs sequence to generate future SIDs, using semantic IDs to generate their textual descriptions). Frequency-weighted GRPO for more diverse predictions are also established by [9], which was cited by the paper. Most notably, the paper's main contributions are quite similar to an uncited recommendation paper [6], from RQ-VAE quantization of items, alignment pretraining with past SID sequence to future SIDs and SIDs to language, followed by Enriched Reasoning with a large teacher model, followed by GRPO. These similarities make the paper too incremental to previous works to be accepted for ICLR's novelty bar.
-
Lack of comparison with more state-of-the-art LLM methods for EHR Diagnosis: While the paper did include methods using Pretrained Language Models' embeddings like BoxLM and HieraMed, the paper excluded significant baselines that can autoregressively model EHR data with LLMs like MERA [10], EHR-R1 [11]. If time and resources are not feasible for exactly reproducing these models, then maybe a light ablation of components similar to the paper's proposal will suffice .
[1] Lu, Chang, et al. "Collaborative graph learning with auxiliary text for temporal event prediction in healthcare." arXiv preprint arXiv:2105.07542 (2021). [2] Yang, Kai, et al. "Kerprint: local-global knowledge graph enhanced diagnosis prediction for retrospective and prospective interpretations." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 4. 2023. [3] Choi, Edward, et al. "GRAM: graph-based attention model for healthcare representation learning." Proceedings of the 23rd ACM SIGKDD international conference on knowledge discovery and data mining. 2017. [4] Xiaorui Su, Shvat Messica, Yepeng Huang, Ruth Johnson, Lukas Fesser, Shanghua Gao, Faryad Sahneh, and Marinka Zitnik. 2025. Multimodal medical code tokenizer. In Proceedings of the 42nd International Conference on Machine Learning (ICML'25), Vol. 267. JMLR.org, Article 2262, 57030–57045. [5] Hang Lv, Kaisong Zhang, Yanchao Tan, and Xing Chen. 2026. Towards Efficient and Interpretable Medical Concept Representation via Ontology-driven Residual Vector Quantization. In Proceedings of the ACM Web Conference 2026 (WWW '26). Association for Computing Machinery, New York, NY, USA, 8645–8648. https://doi.org/10.1145/3774904.3792930 [6] He, Yingzhi, et al. "Reasoning over semantic ids enhances generative recommendation." Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2. 2026. [7] Zheng, Bowen, et al. "Adapting large language models by integrating collaborative semantics for recommendation." 2024 IEEE 40th International Conference on Data Engineering (ICDE). IEEE, 2024. [8] Liu, Zhanyu, et al. "Onerec-think: In-text reasoning for generative recommendation." arXiv preprint arXiv:2510.11639 (2025). [9] Anschel, Oron, et al. "Group-aware reinforcement learning for output diversity in large language models." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025. [10] Ma, Mingyu Derek, et al. "Memorize and rank: Elevating large language models for clinical diagnosis prediction." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 23. 2025. [11] Liao, Yusheng, et al. "Ehr-r1: A reasoning-enhanced foundational language model for electronic health record analysis." arXiv preprint arXiv:2510.25628 (2025).
Questions for authors
Q1 More comparative results with relevant LLM baselines, as stated in Weaknesses, if time and resource allows of course.
Q2 Comparison with the original Qwen 3-1.7B models: How much of the gains in performance are thanks to the original checkpoints' capabilities? Do Qwen3-1.7B fine-tuned on raw ICD codes through the same pipeline without SIDs achieve comparable results?
Q3 Main improvements come from non-headlines contributions: while the paper’s' stated major motivations are the coverage rewards, from the ablation section 4.3 the component that drove the most improvements was Reasoning Activation by the large teacher (from 20.64 to 25.09 in w-F1, Figure 3), while those achieved by coverage rewards are quite incremental (from 23.57 in S4 to 25.41 in S6, Figure 4). I would like the authors to clarify on the relative importance of this component, since as of now most of the gains seem to come from distillation from an already capable teacher with better downstream performance, and not from the paper's main contributions.
Q4 Computational Efficiency of the method: While CARing achieved significant improvements over deep learning and PLM-based baseline, the training and inference cost are substantially larger due to LLM's long reasoning chains, along with heavy multi-step SFT and GRPO. Does the performance gap (around 5-7% from Table 1) justify a significant uptick in computing cost and speed in real clinical scenarios?
Q5 Effectiveness of the reasoning trace: While the paper achieved impressive downstream classification results, the actual quality of the reasoning generated by the models are not evaluated. The interpretability and accuracy of reasoning traces is a critical issue in healthcare, which can greatly affect medical professionals' decision making. Also during multi-positive loss, the method replaces an output diagnosis while retaining the reasoning trajectory that preceded it. Can this technique result in the model’s final predictions being disconnected from its generated reasoning traces?
Q6 Generalization in low-resource datasets: The paper claimed CAring having better generalization capabilities because of MIMIC-IV's 7,000 training sets with many missing codes from test split, however 7,000 are still a relatively large number of samples. What is the performance of the model in even lower resource settings (5, 10, 20, 50…% of the chosen samples)? Will the improvements by training processes such as Reasoning Activation, Coverage Reward… hold under fewer samples? Does performance on rare/new codes suffer substantially?