- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
CardioVascBench is a retrospective benchmark for prioritized diagnosis-list generation from real-world Chinese cardiovascular inpatient records. The dataset comprises 3,262 de-identified episodes from a tertiary cardiology department (2,174 training, 1,088 evaluation), pairing clinical narratives, ECG reports, and other encounter data with expert-adjudicated, ICD-10-standardized discharge diagnoses ordered primary-first; an LLM-generated, structure-preserving English translation of the evaluation set supports aligned cross-lingual comparison.
Critical strengths
- A scarce, released real-world resource in an underserved language and specialty.
- A multi-dimensional evaluation across 36 systems surfaces a concrete, documented blind spot of single-metric evaluation: separating primary-diagnosis identification, coverage, and list precision reveals rank reversals hidden by top-k accuracy alone.
Critical weaknesses
- The sequential generation of multiple diagnoses claimed by this benchmark is not supported by the data; its composite score largely reflects the verbosity of the output.
- This evaluation framework has not been effectively validated.
Questions for authors
-
What is the value of in Eq. 1, and how sensitive are the Table 2/3 rankings to ? Which SapBERT variant is applied to Chinese diagnostic text, directly or after normalization/translation?
-
Exactly which fields are provided as model input? Are the ECG diagnostic-conclusion fields, present in all 1,088 evaluation records per Limitations, and medication/test orders included, and if so, how do you rule out that models copy diagnostic content from the input rather than derive it?
-
If output cardinality is controlled, e.g., all systems prompted to emit the same number of diagnoses, or metrics length-normalized, do the Claude-vs-Lingshu reversal and the +73.5% agent gain persist?
-
Why were 3,261 of the 6,523 valid cases excluded from the released 3,262, and what selection criteria defined the 1,088 high-confidence evaluation cases?
2 comments
Q5+Q6
Q5. Do the ranking reversal and 73.5% agent gain survive length control?
Response: We added paired full-list/first-diagnosis rescoring to Appendix E.1, Tables 9 and 12, and revised Sections 4.3 and 5.2 accordingly. All 39 archived prediction sets were rescored on the same 1,088 cases with identical settings, preserving the original first diagnosis, empty outputs, and complete reference lists. Claude and Lingshu originally averaged 6.30 and 1.50 diagnoses. Their composite scores change from 15.10 versus 28.16 on full lists to 46.15 versus 34.61 after truncation: the original ranking reversal disappears. In the reported stage comparison, the SFT mean rises from 26.39 to 31.86, while the agent mean remains 45.78; the relative gain falls from 73.5% to 43.7%. We now report both results and interpret the original gain as length-dependent. This control is post hoc truncation rather than regeneration under a common single-output prompt; the remaining improvement is system-level and does not isolate graph retrieval.
Q6. Why release only 3,262 cases, and how were the 1,088 evaluation cases selected?
Response: We expanded Section 3.2 and Appendix A.1 to separate ICD-10 eligibility, institutional allocation, and evaluation sampling. Among 8,992 source episodes, 6,713 contained diagnoses, and 6,523 had an exact ICD-10 mapping flag for every recorded diagnosis. The hospital reserved 3,261 eligible episodes for other uses, leaving 3,262 for public release; this was not an additional clinical-quality exclusion. From the 2,328 released episodes with nonempty ECG interpretations, 1,088 were sampled using seed 42 for evaluation, with the remaining 2,174 released episodes assigned to training. These explicit criteria replace the earlier, insufficiently defined “high-confidence” description; no separate clinician-rated confidence threshold was applied. Appendix A.1 also states that the initial institutional partition’s random seed was not logged. Exact mapping and ECG availability constrain the evaluation population, so this procedure does not establish clinical certainty or full cohort representativeness.
Response to Reviewer gr5H
Thank you for the detailed feedback. We have revised the manuscript and added targeted analyses, as detailed below.
Q1. Can sparse references support multi-diagnosis ordering, and does output length dominate the score?
Response: We clarified the task in Section 3.1 and distinguished primary-diagnosis evaluation from multi-reference ordering in Section 4.1. Of the 1,088 evaluation cases, 1,052 have one reference and 36 have multiple references. The former support evaluation against the recorded primary diagnosis but cannot establish ordering among multiple diagnoses. We therefore added a separate analysis in Appendix E.1, Table 11, examining secondary-diagnosis recovery and pair ordering in the 36 multi-reference cases. The single-output agents retrieve no complete diagnosis pairs. We also quantified length sensitivity through first-diagnosis rescoring: Claude scores 46.15 versus Lingshu’s 34.61, reversing their full-list order of 15.10 versus 28.16. Section 4.3 now discusses this dependence on output length and returned-list precision. We consequently limit the interpretation of the aggregate results to agreement with recorded references; plausible additional diagnoses absent from those references remain non-matches under the protocol.
Q2. How was the evaluation framework clinically validated?
Response: We added a clinician-ranking comparison to Section 4.1 and Appendix G.1, Table 13. One clinician reviewed 24 stratified cases, including the clinical materials and recorded discharge references, and ranked three anonymized candidate lists per case. Display order was randomized, and system identities and automatic scores were hidden. Of 72 candidate pairs, both clinician and scorer assigned strict orders to 43: 37 agreed and six disagreed, giving 86.0% conditional agreement with a 95% percentile interval of 72.1%–97.5%. The remaining 29 pairs involved ties. We also report the aggregate system rankings: the clinician placed Claude first, whereas the composite placed Lingshu-7B-AGENT first. This disagreement is reported alongside the pairwise result. The study provides preliminary evidence from one clinician, 24 cases, and three systems; the 86.0% figure applies only to the 43 strictly ordered pairs and does not establish general clinical validity.
Q3. What are β, its ranking sensitivity, and the SapBERT setup for Chinese?
Response: We added the scoring defaults to Section 4.1 and Appendix B.3, Table 5: β=λ=0.7 and K=10. Primary-diagnosis Top-k accuracy uses threshold 0.80; ranked precision, recall, and PrimaryHit use 0.70. Appendix E.1, Table 8, adds sensitivity analysis using archived predictions. Across β=0.3–1.5, up to 23 of the 210 pairs among the 21 systems in Table 2 change order; OpenBioLLM-70B-AGENT remains first. Table 3’s stage ordering persists in these full-list sweeps. Testing λ=0.3 and 1.1 produces no Table 2 pairwise reversals. These results quantify parameter sensitivity without implying invariant rankings. Appendix B.3 also specifies multilingual SapBERT with an XLM-R backbone trained on UMLS 2020AB and normalized [CLS] embeddings. Chinese names undergo whitespace/punctuation normalization and removal of parenthetical suffixes before direct encoding, without translation.
Q4. Which fields are inputs, and can models copy ECG diagnoses?
Response: We expanded Section 3.1 and Appendix B.3 to enumerate the inputs: admission-note title, chief complaint, history of present illness, past medical history, and ECG examination items, findings, and interpretations. Medication and test orders are excluded as separate input fields; discharge diagnoses are reserved for scoring. We also clarified the task as final discharge-diagnosis synthesis after clinical workup. ECG interpretations are deliberately included and may name target conditions, so the task definition does not rule out copying diagnostic terms. Appendix B now explicitly states that ECG-conclusion removal has not been evaluated on the LLMs, while retaining the limitations concerning historical event timestamps and run-specific frozen inputs. Accordingly, we report no measured performance difference for that condition. The revision makes the input specification explicit while identifying dependence on diagnostic wording as an unresolved empirical question.