acceptodds
Under review as a conference paper at ICLR 2027

CardioVascBench: Prioritized Diagnosis Generation from Real-World Chinese Cardiovascular Records

Abstract

Identifying the primary diagnosis alone does not fully characterize a model’s diagnosis-generation performance. We introduce CardioVascBench, a retrospective benchmark for prioritized diagnosis generation from real-world Chinese cardiovascular inpatient records. It contains 3,262 de-identified cases, with 2,174 for training and 1,088 for evaluation, linking clinical narratives and textual ECG reports to expert-verified, ordered discharge diagnoses standardized to ICD-10. An English translation of the evaluation set supports aligned cross-lingual comparison. Our evaluation protocol combines terminology normalization and rank-aware scoring to separately assess primary-diagnosis identification, reference-diagnosis coverage, and list precision. Evaluation across 36 systems shows that higher primary-diagnosis accuracy does not necessarily imply higher list precision against discharge references. We also provide CardioKG-Agent, a knowledge-enhanced baseline combining entity normalization, constrained graph retrieval, and guideline evidence. In a staged comparison of three baseline models, supervised fine-tuning plus the agent achieves a 73.5% higher mean composite score than fine-tuning alone, representing a system-level gain. These results measure retrospective agreement with recorded diagnoses; agreement between metric rankings and physician judgments remains to be validated. CardioVascBench supports research on diagnosis prioritization and knowledge-enhanced generation.

Primary area:
datasets and benchmarks
Keywords:
Medical Benchmarking, Large Language Models, Cardiovascular Diagnosis, Clinical Natural Language Processing, Chinese Clinical Records, Knowledge-Augmented Agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gr5HOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

CardioVascBench is a retrospective benchmark for prioritized diagnosis-list generation from real-world Chinese cardiovascular inpatient records. The dataset comprises 3,262 de-identified episodes from a tertiary cardiology department (2,174 training, 1,088 evaluation), pairing clinical narratives, ECG reports, and other encounter data with expert-adjudicated, ICD-10-standardized discharge diagnoses ordered primary-first; an LLM-generated, structure-preserving English translation of the evaluation set supports aligned cross-lingual comparison.

Critical strengths
  • A scarce, released real-world resource in an underserved language and specialty.
  • A multi-dimensional evaluation across 36 systems surfaces a concrete, documented blind spot of single-metric evaluation: separating primary-diagnosis identification, coverage, and list precision reveals rank reversals hidden by top-k accuracy alone.
Critical weaknesses
  • The sequential generation of multiple diagnoses claimed by this benchmark is not supported by the data; its composite score largely reflects the verbosity of the output.
  • This evaluation framework has not been effectively validated.
Questions for authors
  • What is the value of β\beta in Eq. 1, and how sensitive are the Table 2/3 rankings to β\beta? Which SapBERT variant is applied to Chinese diagnostic text, directly or after normalization/translation?

  • Exactly which fields are provided as model input? Are the ECG diagnostic-conclusion fields, present in all 1,088 evaluation records per Limitations, and medication/test orders included, and if so, how do you rule out that models copy diagnostic content from the input rather than derive it?

  • If output cardinality is controlled, e.g., all systems prompted to emit the same number of diagnoses, or metrics length-normalized, do the Claude-vs-Lingshu reversal and the +73.5% agent gain persist?

  • Why were 3,261 of the 6,523 valid cases excluded from the released 3,262, and what selection criteria defined the 1,088 high-confidence evaluation cases?

2 comments
AuthorsOpenReview ↗

Response to Reviewer gr5H

Thank you for the detailed feedback. We have revised the manuscript and added targeted analyses, as detailed below.

Q1. Can sparse references support multi-diagnosis ordering, and does output length dominate the score?

Response: We clarified the task in Section 3.1 and distinguished primary-diagnosis evaluation from multi-reference ordering in Section 4.1. Of the 1,088 evaluation cases, 1,052 have one reference and 36 have multiple references. The former support evaluation against the recorded primary diagnosis but cannot establish ordering among multiple diagnoses. We therefore added a separate analysis in Appendix E.1, Table 11, examining secondary-diagnosis recovery and pair ordering in the 36 multi-reference cases. The single-output agents retrieve no complete diagnosis pairs. We also quantified length sensitivity through first-diagnosis rescoring: Claude scores 46.15 versus Lingshu’s 34.61, reversing their full-list order of 15.10 versus 28.16. Section 4.3 now discusses this dependence on output length and returned-list precision. We consequently limit the interpretation of the aggregate results to agreement with recorded references; plausible additional diagnoses absent from those references remain non-matches under the protocol.

Q2. How was the evaluation framework clinically validated?

Response: We added a clinician-ranking comparison to Section 4.1 and Appendix G.1, Table 13. One clinician reviewed 24 stratified cases, including the clinical materials and recorded discharge references, and ranked three anonymized candidate lists per case. Display order was randomized, and system identities and automatic scores were hidden. Of 72 candidate pairs, both clinician and scorer assigned strict orders to 43: 37 agreed and six disagreed, giving 86.0% conditional agreement with a 95% percentile interval of 72.1%–97.5%. The remaining 29 pairs involved ties. We also report the aggregate system rankings: the clinician placed Claude first, whereas the composite placed Lingshu-7B-AGENT first. This disagreement is reported alongside the pairwise result. The study provides preliminary evidence from one clinician, 24 cases, and three systems; the 86.0% figure applies only to the 43 strictly ordered pairs and does not establish general clinical validity.

Q3. What are β, its ranking sensitivity, and the SapBERT setup for Chinese?

Response: We added the scoring defaults to Section 4.1 and Appendix B.3, Table 5: β=λ=0.7 and K=10. Primary-diagnosis Top-k accuracy uses threshold 0.80; ranked precision, recall, and PrimaryHit use 0.70. Appendix E.1, Table 8, adds sensitivity analysis using archived predictions. Across β=0.3–1.5, up to 23 of the 210 pairs among the 21 systems in Table 2 change order; OpenBioLLM-70B-AGENT remains first. Table 3’s stage ordering persists in these full-list sweeps. Testing λ=0.3 and 1.1 produces no Table 2 pairwise reversals. These results quantify parameter sensitivity without implying invariant rankings. Appendix B.3 also specifies multilingual SapBERT with an XLM-R backbone trained on UMLS 2020AB and normalized [CLS] embeddings. Chinese names undergo whitespace/punctuation normalization and removal of parenthetical suffixes before direct encoding, without translation.

Q4. Which fields are inputs, and can models copy ECG diagnoses?

Response: We expanded Section 3.1 and Appendix B.3 to enumerate the inputs: admission-note title, chief complaint, history of present illness, past medical history, and ECG examination items, findings, and interpretations. Medication and test orders are excluded as separate input fields; discharge diagnoses are reserved for scoring. We also clarified the task as final discharge-diagnosis synthesis after clinical workup. ECG interpretations are deliberately included and may name target conditions, so the task definition does not rule out copying diagnostic terms. Appendix B now explicitly states that ECG-conclusion removal has not been evaluated on the LLMs, while retaining the limitations concerning historical event timestamps and run-specific frozen inputs. Accordingly, we report no measured performance difference for that condition. The revision makes the input specification explicit while identifying dependence on diagnostic wording as an unresolved empirical question.

Q5. Do the ranking reversal and 73.5% agent gain survive length control?

Response: We added paired full-list/first-diagnosis rescoring to Appendix E.1, Tables 9 and 12, and revised Sections 4.3 and 5.2 accordingly. All 39 archived prediction sets were rescored on the same 1,088 cases with identical settings, preserving the original first diagnosis, empty outputs, and complete reference lists. Claude and Lingshu originally averaged 6.30 and 1.50 diagnoses. Their composite scores change from 15.10 versus 28.16 on full lists to 46.15 versus 34.61 after truncation: the original ranking reversal disappears. In the reported stage comparison, the SFT mean rises from 26.39 to 31.86, while the agent mean remains 45.78; the relative gain falls from 73.5% to 43.7%. We now report both results and interpret the original gain as length-dependent. This control is post hoc truncation rather than regeneration under a common single-output prompt; the remaining improvement is system-level and does not isolate graph retrieval.

Q6. Why release only 3,262 cases, and how were the 1,088 evaluation cases selected?

Response: We expanded Section 3.2 and Appendix A.1 to separate ICD-10 eligibility, institutional allocation, and evaluation sampling. Among 8,992 source episodes, 6,713 contained diagnoses, and 6,523 had an exact ICD-10 mapping flag for every recorded diagnosis. The hospital reserved 3,261 eligible episodes for other uses, leaving 3,262 for public release; this was not an additional clinical-quality exclusion. From the 2,328 released episodes with nonempty ECG interpretations, 1,088 were sampled using seed 42 for evaluation, with the remaining 2,174 released episodes assigned to training. These explicit criteria replace the earlier, insufficiently defined “high-confidence” description; no separate clinician-rated confidence threshold was applied. Appendix A.1 also states that the initial institutional partition’s random seed was not logged. Exact mapping and ECG availability constrain the evaluation population, so this procedure does not establish clinical certainty or full cohort representativeness.

Reviewer_PZMwOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces CardioVascBench, a retrospective benchmark of 3,262 de-identified cardiology inpatient records from a Chinese tertiary hospital (2,174 train / 1,088 eval). Input data includes admission notes, history, labs, and textual ECG reports; the target consists of ICD-10 mapped discharge diagnoses. Ground truth labels were created via an ensemble matcher, refined by multi-LLM voting, and adjudicated by cardiologists. The paper evaluates 36 systems using a customized metric (nDCG×(0.7+0.3⋅PrimaryHit)×PrecisionβnDCG \times (0.7 + 0.3 \cdot \text{PrimaryHit}) \times \text{Precision}^\beta) and proposes CardioKG-Agent, a knowledge-graph and guideline-augmented multi-LLM pipeline that reports a 73.5% improvement over standard SFT.

Critical strengths
  • Niche Clinical Domain Data: Provides a rare public dataset of real-world Chinese inpatient narratives and discharge diagnoses from a cardiology department, accompanied by IRB approval and plans for public release.
  • Candid Limitation Reporting: The authors explicitly acknowledge multiple dataset, metric, and pipeline limitations throughout the manuscript.
Critical weaknesses
  1. Severe Dataset Defects & Label Quality Issues:
    • Missing Multi-Label/Ranking Structure: Although framed around prioritized diagnosis lists, 97% of evaluation cases (1,052 / 1,088) contain only a single ground-truth diagnosis (mean 1.09), invalidating ranking and recall metrics over list positions.
    • Noisy & Contaminated Label Space: Labels mix symptoms with actual diagnoses (e.g., R00–R99 symptoms like chest pain account for 28.8% of primary diagnoses), contain near-duplicates (e.g., "Coronary Heart Disease" vs. "Coronary Atherosclerotic Heart Disease"), and include non-cardiovascular noise (e.g., glaucoma, post-cholecystectomy).
    • Circular Annotations & Data Leakage: Ground-truth labels were generated using LLM ensembles that include GPT-5 and Claude-4.5 (matching evaluated model families). Furthermore, patient-level separation between train and eval splits cannot be verified, inputs contain explicit ECG diagnostic text (which directly reveals target diagnoses like arrhythmias), and LLM-based English translations lack quality validation.
  2. Unreliable Evaluation & Artifact-Driven Metric Gains:
    • Non-Reproducible Protocol: The authors openly disclose in Appendix B that per-run mappings to frozen inputs are unestablished, prompt instructions contradict reported @10 table configurations, and composite score hyperparameters (β,λ\beta, \lambda) are omitted.
    • Length-Artifact Inflation: Because evaluation cases overwhelmingly feature only 1 true label, Precision@10 (and thus the composite score) is heavily penalized if a model outputs multiple diagnoses. The reported 73.5% agent gain and model "ranking reversals" appear to be artifacts of output length: agents outputting a single diagnosis artificially inflate Precision@10 relative to models outputting longer candidate lists.
Questions for authors
  1. Protocol Standardization & Length Control:
    • Can you re-evaluate all systems under a single, frozen, open-sourced configuration with a fixed output candidate budget (e.g., evaluating top-1 prediction only or enforcing k=1k=1 output) and explicit β/λ\beta/\lambda hyperparameters?
    • What is the average number of output diagnoses per system? Do the core findings (e.g., the 73.5% agent gain and Claude vs. Lingshu ranking reversal) hold when output length is strictly controlled?
  2. Input Leakage & Patient Splitting:
    • How do model performances change if textual ECG diagnostic conclusions are removed from the input?
    • Can you provide verifiable confirmation of strict, patient-level separation between the training and evaluation splits?
  3. Label Noise & Circularity:
    • How were symptom-level, non-cardiovascular, and near-duplicate labels handled during evaluation?
    • What was the inter-annotator agreement (e.g., Fleiss' κ\kappa) among cardiologists during label adjudication? Were evaluated model families (or their immediate predecessors) involved in creating the evaluation set labels?
  4. Agent Ablations & KG Construction:
    • How was the 5,234-entity Knowledge Graph constructed, and was any data from the evaluation split used during its creation?
    • Can you provide a controlled ablation study isolating the individual contributions of text RAG, KG retrieval, guideline retrieval, and additional inference compute?
3 comments
AuthorsOpenReview ↗

Response to Reviewer PZMw

Thank you for the detailed assessment. We respond to each concern and question below.

Q1. Can mostly single-reference cases support recall and diagnosis-ordering evaluation?

Response: Section 4.1 now distinguishes single-reference agreement from ordering multiple diagnoses. Recall and nDCG remain computable with one reference, but their interpretation is limited to agreement with that recorded diagnosis under the specified matching rule. The 1,052 single-reference cases cannot validate relative ordering between primary and secondary conditions. We therefore added a separate analysis of the 36 multi-reference cases in Appendix E.1, Table 11, covering 93 recorded secondary diagnoses and 198 within-case reference pairs. It separately reports secondary recovery, joint pair retrieval, and ordering conditional on retrieving both members. The single-output agents retrieve no complete pairs, so these results do not establish their superiority in secondary-diagnosis ordering. The small multi-reference subset also limits the strength of broader ordering conclusions.

Q2. Why retain symptom, non-cardiovascular, and near-duplicate labels?

Response: Appendix A.1 now explains that references preserve documented hospital discharge labels, including symptoms and non-cardiovascular conditions, under the same ICD-10 eligibility rule. Symptom/sign codes account for 28.8% of primary references, corresponding to 313 of 1,088 cases. We preserve original names alongside standardized ICD names and codes, and the within-case audit found no repeated exact names or main codes. However, this does not establish complete clinical synonym harmonization or correctness of every atypical mapping. The two coronary-disease names identified by the reviewer remain separate recorded labels. SapBERT alias matching and one-to-one assignment address prediction–reference matching but do not merge the reference vocabulary. The revised description therefore makes the label scope explicit while retaining concept-level granularity and mapping quality as limitations.

Q3. How do you address annotation circularity, leakage, and translation quality?

Response: We revised Section 3.2 and Appendix A.1 to distinguish hospital diagnoses, screening, and adjudication. GPT-5 and Claude-4.5 screened ambiguous strings or ICD candidates; flagged entries received final adjudication from one clinically experienced coauthor. All 1,181 evaluation references trace to source entries marked exact, but this flag does not exclude upstream screening bias. The patient-ID audit establishes zero overlap between 2,174 training IDs and 1,088 evaluation IDs. Section 3.1 and Appendix B explicitly include ECG interpretations, and no conclusion-removal experiment has quantified dependence on their diagnostic wording. Appendix A.1 documents expert review of the English translation, without a quantitative translation-error study. The revision thus documents patient separation and clarifies label provenance while explicitly retaining the unresolved input-dependence, screening-bias, and language-protocol limitations.

Q4. How reproducible is the evaluation given missing historical inputs and settings?

Response: Section 4.1 and Appendix B now separate generation length from scoring cutoff K=10 and disclose β=0.7, λ=0.7, ranked threshold 0.70, and primary Top-k threshold 0.80. Appendix F reproduces retained prompts, while Appendix E.1 reports output lengths and fixed-configuration rescoring of 39 archived Chinese prediction sets. These changes make the scoring procedure explicit but do not reconstruct the original generation runs. Per-run frozen inputs, prompt bindings, exact model/API revisions, access dates, and decoding/retry settings could not be fully recovered. An anonymized package containing the prediction index, scoring code, and offline replay script has not yet been released. We therefore distinguish the completed offline analyses and documented settings from the unresolved reproducibility of the original generation runs.

Q5. Are the agent gains and ranking reversals artifacts of output length?

Response: The added control confirms substantial length sensitivity. Appendix E.1 compares each archived response with its first diagnosis under identical scoring settings. Claude’s composite score changes from 15.10 to 46.15, while Lingshu’s changes from 28.16 to 34.61; the original Claude–Lingshu composite reversal therefore disappears. In Table 9, the reported SFT-group mean rises from 26.39 to 31.86, whereas the agent-group mean stays at 45.78 because the three agents already return one diagnosis per case. Their relative difference falls from 73.5% to 43.7%. Sections 4.3 and 5.2 now report both results. The remaining positive difference is a system-level comparison under a one-candidate scoring cap. It does not isolate KG retrieval, equalize inference compute, or establish performance under identically constrained new generation.

Q6. Were all systems reevaluated under a frozen, open configuration with controlled output length?

Response: We standardized the scoring stage for 39 archived Chinese prediction sets covering the same 1,088 cases. Table 5 specifies K=10, β=λ=0.7, ranked threshold 0.70, and primary Top-k threshold 0.80. Appendix E.1 scores each saved response as a full list and after retaining only its first diagnosis, preserving empty outputs and all references. This supplies a common one-candidate evaluation cap, but we have not regenerated every system under one frozen single-output prompt. Appendix F documents the retained templates and missing historical bindings. The prediction index, scoring code, and offline replay script have not yet been released as an anonymized package. Thus, fixed-configuration rescoring is complete; the current results do not establish a uniformly generated and fully released end-to-end experiment.

Q7. What are the output lengths, and which findings survive a fixed candidate count?

Response: Table 12 now reports mean, median, range, and output-count bins for all 39 archived prediction sets, alongside paired full-list and first-diagnosis scores. Claude averages 6.30 diagnoses, Lingshu 1.50, and each of the three agents 1.00. The headline findings do not survive unchanged: after retaining only the first diagnosis, Claude scores 46.15 versus Lingshu’s 34.61, eliminating the original composite reversal. The agent-group advantage over the reported SFT group decreases from 73.5% to 43.7% in Table 9. Sections 4.3 and 5.2 discuss this change. Because the control truncates existing responses, it estimates sensitivity to candidate count without showing how fresh outputs would change under common single-diagnosis instructions or matched inference budgets. The remaining advantage is consequently narrower than the original full-list claim.

Q8. How does performance change after removing ECG diagnostic conclusions?

Response: We have not measured that performance difference. Section 3.1 and Appendix B now specify the ECG input fields and explicitly disclose that conclusion-removal experiments have not been conducted on the evaluated LLMs. The benchmark includes ECG interpretations as intermediate report information for retrospective discharge-diagnosis generation. Removing only the conclusion field would still leave the clinical narrative, ECG examination items, and findings available; it would not create a history-only task. The current offline reanalysis cannot quantify this change because it reuses outputs generated with the original inputs. Consequently, dependence on diagnostic wording and possible copying remain unresolved. The stated clinical decision point explains the input choice but does not itself demonstrate that models infer diagnoses independently of the supplied conclusions.

Q9. What evidence establishes patient-level separation between training and evaluation?

Response: Section 3.2 and Appendix B describe one episode per stable anonymized patient identifier before splitting. We additionally report the source-to-release identifier check here: the hospital uses the same identifier for records from one patient, and the released identifier fields are compared as strings. Training contains 2,174 unique IDs and evaluation contains 1,088, with zero intersection; every released ID is traceable to the structured source cohort of 8,992 unique patient IDs. This directly supports patient separation in the released cohort. Appendix B additionally reports no exact duplicates in the normalized admission/ECG text and no cross-split pairs above 0.90 character 5-gram Jaccard similarity. Those text-based checks provide supplementary evidence; the patient-level conclusion is based on the identifier audit and is limited to the released hospital cohort.

Q10. How are symptom, non-cardiovascular, and near-duplicate labels scored?

Response: Appendix A.1 and Appendix B.3 now make the inclusion and scoring rules explicit. Symptom, cardiovascular, and non-cardiovascular diagnoses are retained and scored against their recorded references under the same rules. Each reference stores its original Chinese name, standardized ICD-10 name, and code. After text normalization, prediction similarity is the maximum SapBERT similarity against the recorded name and available ICD alias. Both reported ranked-assignment variants use each reference at most once, preventing repeated credit for that entry; the variants differ in whether assignment first requires the ranked threshold. The within-case audit found no duplicate exact names or main codes. These measures address matching and repeated assignments but do not merge the reference vocabulary or prove that all clinical near-synonyms and questionable mappings have been resolved.

Q11. Was annotation agreement measured, and did evaluated model families assist labeling?

Response: No inter-annotator agreement was measured, and evaluated model families did participate in screening. Section 3.2 and Appendix A.1 now specify that GPT-5 and Claude-4.5, among other models, screened ambiguous diagnosis strings or candidate ICD mappings. The clinical labels originated from hospital discharge records, with flagged entries receiving final adjudication from one clinically experienced coauthor. Because the workflow did not collect independent ratings from multiple clinicians on the same entries, it cannot supply Fleiss’ κ or an equivalent agreement estimate. Distinguishing screening from final adjudication does not establish that overlapping model families had no influence on which entries were reviewed. The separate single-clinician ranking experiment in Appendix G.1 evaluates candidate-list preferences and does not provide evidence of independent label-annotation agreement.

Q12. How was the knowledge graph constructed, and were evaluation data excluded?

Response: Appendix C now identifies the construction sources and exclusion scope of the 5,234-entity graph. Its sources are the Chinese Guidelines for Cardiovascular Disease Risk Assessment and Management and three cardiology texts: New Differential Diagnosis of Cardiovascular Diseases; Classic and Complex Cases in Cardiovascular Disease from Beijing Friendship Hospital; and Clinical Case Consultations and Commentary, Cardiovascular Medicine Volume. Clinical concepts and relationships from these sources were organized into typed entities and standardized relations, including diseases, symptoms, tests, drugs, and treatments. The complete evaluation split, including both case records and reference diagnoses, was excluded from graph construction. Appendix C also describes the subsequent constrained one-to-two-hop retrieval procedure. These additions make the external knowledge sources, representation, and exclusion of evaluation data explicit.

Q13. Do the ablations isolate retrieval components from additional inference compute?

Response: Appendix C, Table 6, adds five Lingshu-7B configurations on the same 1,088 cases, each requesting one primary diagnosis. Top-1 accuracy is 24.72% without retrieval, 36.27% with text RAG, 36.55% with KG retrieval, 41.06% with guidelines, and 43.93% with combined retrieval. The table also reports composite scores and actual calls and tokens, including retries. Text RAG and KG use the same source corpus, and the four multistage variants share inference limits and an evidence-token cap. Their mean calls nevertheless differ: 4.3, 5.8, 4.1, and 8.9, versus one for the single-pass baseline. There is no compute-matched multistage no-retrieval control. These comparisons therefore characterize system configurations; the independent contribution of additional inference compute remains unidentified, as reflected in Section 5.2’s system-level interpretation.

Reviewer_bhUnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

CardioVascBench presents a benchmark for ranked diagnoses for Chinese cardiovascular real-world cases from clinical narratives and textual ECG reports. It is built from 3,262 expert-reviewed valid cases (2,174 for training and 1,088 for evaluation) out of 6,523 valid episodic cases from one Chinese hospital. It used a multi-method ICD-10 mapping ( e.g., SapBERT, fuzzy, and rule matching) to standardize discharged diagnoses. Multi-dimensional evaluation, including nDCG@10, precision, recall, top-k primary accuracy, and a composite score, was performed over 36 systems. The proposed knowledge-graph-guided agent, enhanced with fine-tuning (CardioKG-Agent), reportedly increases the mean composite score by 73.5% compared with the fine-tuned-only baseline.

Critical strengths

The benchmark addresses a timely and practical problem by evaluating models on real data from Chinese inpatient records. The focused scope on cardiovascular is reasonable and clear.

Evaluating multiple metrics, such as precision, recall, and ranking measures, across 36 different systems is substantial. It makes the differences between models clear and provides a useful reference for future work. Importantly, the evaluations exposes that primary diagnosis accuracy and list precision rank models differently, which a single score would hide.

Critical weaknesses

The paper presents a prioritized diagnoses list, but the evaluation does not provide enough evidence to support ordered list generation. According to Appendix A.2, 1,052 of 1,088 evaluation cases (~97%) include one reference entry, 12 have two, and 24 have three or more (mean 1.09, max 5). With a single relevant item, this evaluation set can test whether the primary diagnosis ranks highly, but it cannot support ordering primary and secondary diagnoses. The average diagnoses per episode is 7,786/6,713 ≈ 1.16, which seems low for tertiary-cardiology discharge summaries that usually record multiple health conditions. Therefore, the structured table mostly captures the primary diagnosis, and possible correct secondary diagnoses are scored as false positives.

The dataset construction and split are ambiguous. They report that 6,523 cases were valid after cleaning but only release 3,262. The selection procedure and the meaning of “high-confidence” cases are not specified. The high-confidence selected cases might over-represent common and clearly mappable labels. As the authors mention, the patients’ separation in the dataset cannot be confirmed, and if the train-test split is not patient-level, any patient overlap may constitute train-test leakage in SFT training and limit confidence in the results.

The more diagnoses a model produces may negatively affect its ranking. The average reference diagnoses per case is about 1.09; therefore, a model that generates several diagnoses may receive low precision because some diagnoses are absent from the reference list. This directly affects the composite score, which may explain why Claude-sonnet-4.5 has the highest primary diagnosis accuracy but a lower composite score than the agent-enhanced systems. Also, the agents have the same Top-1 and Top-3 accuracy, which may mean they produce short lists or that their additional predictions do not recover any missed primary diagnoses. Furthermore, Equation 5 independently matches each generated diagnosis to its best reference. Therefore, several predictions could receive credit for matching the same reference diagnosis without generating any additional diagnoses. It is not clear how this work prevents this.

Questions for authors
  1. Do the reference lists include all recorded discharge diagnoses, including secondary conditions, or have they been selected? Why do approximately 97% of evaluation cases contain only one entry?
  2. Can you report performance separately on cases with multiple distinct reference diagnoses? Please clarify if the reference entries are deduplicated and if not, please explain the reason. And what do the results support about ordering primary and secondary diagnoses?
  3. Why were only the 3,262 released cases chosen from the 6,523 valid cases? What does “high-confidence” or “exact match” exactly mean? How is this selection rule applied differently to training and evaluation cases?
  4. How does the work handle repeated patients, duplicate records, or overlapping notes across training and evaluation? Without patient-level separation, how should the SFT results be interpreted?
  5. How many diagnoses does each system generate on average, and what is the distribution of output lengths for each system? How do the system scores change if you enforce the same output count or truncate the current outputs to a fixed count?
  6. How is Precision@10 calculated when a model generates fewer than ten diagnoses? Can multiple predictions receive credit for matching the same reference, and how does this affect other metrics?
  7. What are the hyperparameter values, such as β, λ, and k?
  8. How was the Chinese-output instruction handled for English evaluation?
  9. Does the agent use the same output schema and item limit as the baselines? Which prompt and output language were used for the English evaluation set, given the “Output in Chinese” instruction? Please provide the Tool, Expert, and General model prompts
3 comments
AuthorsOpenReview ↗

Response to Reviewer bhUn

Thank you for the detailed comments and questions. We address each point individually below.

Q1. What can mostly single-reference cases establish about ordered diagnosis generation?

Response: We added an explicit distinction between single-reference evaluation and multi-reference ordering in Section 4.1, together with a separate analysis of the 36 multi-reference cases in Appendix E.1, Table 11. Appendix A.2 now reports the source-to-export check: all 1,052 single-reference evaluation cases already contained one diagnosis in the hospital table. The evaluation mean of 1.09 and source mean of 1.16 describe recorded entries, rather than patients’ complete comorbidity burden. Section 5.1 also states the limited support for multi-diagnosis conclusions. The aggregate results should therefore primarily be interpreted as agreement with sparse recorded diagnoses; the small multi-reference analysis does not establish broad secondary-diagnosis ordering ability. Section 3.1 further clarifies that a plausible additional diagnosis absent from the discharge reference is a non-match under this protocol, rather than an established clinical error.

Q2. How were cases selected and patient separation established?

Response: We expanded Section 3.2 and Appendix A.1 to separate eligibility, institutional release, and evaluation sampling. The same cleaning and exact-ICD-mapping eligibility rule was applied before partitioning the 6,523 eligible episodes. A random institutional partition reserved 3,261 episodes for other hospital uses and allocated 3,262 to release; its initial shuffle seed was not logged. Evaluation then sampled 1,088 of the 2,328 released episodes with nonempty ECG interpretations using seed 42, leaving 2,174 for training. We also updated Appendix B with the patient-ID check: initial cleaning retained one episode per patient, and the released splits have zero patient-ID intersection. These additions clarify the selection steps and document patient separation. They do not remove selection effects from exact mapping and ECG availability, which can favor more readily standardized and more fully documented cases.

Q3. Do output length and repeated matching explain the model rankings?

Response: We revised Section 4.3 and Appendix E.1 to report observed output lengths and a paired first-diagnosis control. Each archived agent returns exactly one diagnosis, explaining its identical Top-1 and Top-3 accuracy. The control confirms length dependence: Claude’s composite score changes from 15.10 to 46.15 and Lingshu’s from 28.16 to 34.61, so their full-list ordering does not persist. Separately, Appendix B.3 corrects the incomplete Eq. 5 description by specifying greedy one-to-one assignments: each reference can contribute to only one ranked assignment. We also added threshold-qualified rescoring in Table 10. Unlike the archived rule, it prevents a sub-threshold prediction from consuming a reference needed by a later stronger match. The main tables retain the archived scorer, and the new variant is reported separately. These analyses address length sensitivity and assignment behavior without claiming a new generation experiment under a common output budget.

Q4. Are all discharge diagnoses retained, and why are most references singletons?

Response: We clarified the export rule in Appendix A.2 and compared the reference list for each of the 1,088 evaluation cases with its source diagnosis array. The released list equals the first five recorded entries in their original order; therefore, it is complete only when the source has at most five diagnoses. The cap affects 11 episodes with longer source lists. All 1,052 single-reference evaluation cases already had one entry in the source table, so the five-entry cap does not create the approximately 97% singleton share. We also clarify that the Top-20 diagnosis table is a frequency summary, not a vocabulary restriction. These checks establish how recorded entries were exported; they do not establish that the hospital’s structured diagnosis table exhaustively captures every clinically valid secondary condition.

Q5. How do models perform on multi-reference cases, and are references deduplicated?

Response: We added separate results for the 36 multi-reference cases in Appendix E.1, Table 11, including nDCG, recall, secondary-diagnosis recovery, joint retrieval of reference pairs, and ordering conditional on retrieving both members. Appendix A.2 also reports the duplicate audit. We preserved recorded entries and their order, subject to the five-entry cap, without additional concept-level merging. No duplicate exact names or main ICD-10 codes occurred within a case, so none required removal under those criteria; this does not establish that all clinical near-synonyms represent distinct concepts. The subset analysis uses threshold-qualified one-to-one matching. The single-output agents retrieve no complete reference pairs and therefore provide no evidence of superior secondary-diagnosis ordering. Conclusions about multi-diagnosis ordering are limited to these 36 cases and the diagnoses actually recorded.

Q6. What do “high-confidence” and “exact match” mean for each split?

Response: Section 3.2 and Appendix A.1 now specify the common eligibility rule: an episode must contain a recorded diagnosis, and every recorded diagnosis must carry an exact ICD-10 mapping flag. This flag-based rule applies before partitioning, so training and evaluation share the same ICD eligibility criterion. Of 6,523 eligible episodes, 3,261 were randomly reserved for hospital use and 3,262 allocated to release. Evaluation additionally required a nonempty ECG interpretation before seed-42 sampling. “High-confidence” therefore does not denote a separate evaluation-only clinical confidence threshold. Here, “exact” refers to the mapping category recorded by the curation pipeline; it is not an estimate of diagnostic correctness or a guarantee of independence from upstream LLM assistance. Mapping eligibility and ECG availability can still restrict representativeness, and the revised description makes these selection conditions explicit.

Q7. How were repeated patients and overlapping notes handled before evaluating SFT?

Response: Section 3.2 and Appendix B now document one-episode-per-patient curation and patient-disjoint splits, while Section 6 limits interpretation to the same-hospital setting. We additionally report the source-to-release identifier check here: the same patient uses the same anonymized ID, and evaluation export preserves the source identifier. The source contains 8,992 unique patient IDs; all 2,174 training IDs and 1,088 evaluation IDs are present in that source, with zero intersection between the splits. Appendix B also reports no identical normalized admission/ECG texts and no cross-split character 5-gram Jaccard similarity above 0.90. Patient separation is supported by the identifier check, while the text audit provides supplementary evidence. Accordingly, the SFT results concern held-out patients within this hospital; they do not establish generalization to patients from other hospitals.

Q8. What are the output-length distributions and scores after fixed-count truncation?

Response: We added per-system length distributions and paired rescoring to Appendix E.1, Table 12, for 39 archived Chinese prediction sets on the same 1,088 cases. The table gives the mean, median, range, and counts for 0, 1, 2–5, 6–10, and more than 10 diagnoses. Claude averages 6.30 diagnoses, with median 6 and range 1–19; Lingshu-7B averages 1.50, with median 1 and range 1–7. The three archived agents each return one diagnosis. Under unchanged scoring settings and references, retaining only the first diagnosis changes Claude’s composite from 15.10 to 46.15 and Lingshu’s from 28.16 to 34.61, reversing their full-list order. Empty outputs remain empty. These are paired post hoc truncations of existing responses; we have not regenerated all systems under a common forced output count.

Q9. How is Precision@10 computed, and can one reference receive repeated credit?

Response: We made the denominator and assignment rule explicit in Section 4.1 and Appendix B.3. Returned-list Precision@10 equals m/min(10,n), where n is the returned-list length and m counts distinct ranked assignments with similarity at least 0.70 within the first ten predictions; empty outputs score zero. Thus, a list shorter than ten is divided by its actual length. Each reference can be assigned at most once, so repeated paraphrases cannot receive repeated ranked credit. We also added Table 10 to distinguish the archived positive-similarity assignment from the threshold-qualified variant, which leaves references available after a sub-threshold prediction. In this reanalysis, removing partial credit lowered nDCG, while precision and recall increased for some systems. PrimaryHit and Top-k accuracy are computed directly against the primary reference, independently of ranked assignments, and remain unchanged.

Q10. What values are used for β, λ, and k?

Response: We added the fixed scoring configuration to Section 4.1 and Appendix B.3, Table 5: the precision exponent is β=0.7, reference-priority decay is λ=0.7, and ranked cutoff is K=10. This uppercase K is distinct from the k in primary-diagnosis Top-k accuracy, which uses k∈{1,2,3}. Primary-diagnosis Top-k accuracy applies a similarity threshold of 0.80, whereas ranked precision, recall, and PrimaryHit use 0.70. Appendix E.1, Table 8, additionally reports one-at-a-time sensitivity analyses for weights and matching thresholds. For example, varying β from 0.3 to 1.5 reverses up to 23 of the 210 system pairs in Table 2. These additions disclose the defaults and quantify dependence on the tested parameter choices.

Q11. How was the Chinese-output instruction handled during English evaluation?

Response: We revised Appendix F to distinguish translated input materials from the output language of a historical generation run. It now explicitly states that the retained English prompt still requests Chinese diagnosis names and that the exact prompt/output-language bindings of historical English runs could not be verified. The additional rescoring in Appendix E.1 therefore uses Chinese predictions and Chinese references only. Separately, Section 3.2 and Appendix A.1 describe the English data construction: translation of the same 1,088 cases while preserving structure, diagnosis order, and ICD-10 codes, with ICD-normalized English labels and expert review. These additions document the available translation and template materials. They do not establish how every historical English run handled the Chinese-output instruction or demonstrate a standardized cross-language generation protocol.

Q12. Do agents share baseline output constraints, and which prompts were used?

Response: Appendix F.3 now provides the retained Tool, Expert, and General prompts, covering entity extraction, candidate generation and selection, and evidence-guided synthesis with JSON formatting. Both baseline and agent return a JSON diagnosis array with one name per item. Their count instructions cannot be certified identical: the baseline requests a Chinese primary diagnosis, while the agent formatter permits up to five diagnoses and defaults to one; its parser deduplicates names in order and retains at most ten. These generation/parser limits differ from scoring cutoff K=10. Table 12 confirms one diagnosis per case in the archived agent outputs. The retained English prompt still requests Chinese diagnosis names. Historical prompt/output-language bindings and helper-model settings could not be fully recovered, so Appendix F documents available templates without claiming uniform historical generation constraints or a verified English-output protocol.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revise Summary

We thank the reviewers for their feedback. We clarified the task and data protocol and added analyses of scoring, clinician rankings, and retrieval.

A scarce clinical resource. CardioVascBench releases 3,262 de-identified inpatient episodes from a Chinese tertiary cardiology department. It pairs routinely documented Chinese admission notes and available textual ECG reports with ordered, ICD-10-standardized discharge diagnoses. Public benchmarks with this combination of authentic records, language, specialty, and ranked diagnosis labels are scarce, as reviewers gr5H and PZMw noted. An expert-reviewed English translation supports comparisons on the same evaluation cases.

Final diagnosis and the ECG conclusion. CardioVascBench evaluates prioritized final inpatient diagnosis after the diagnostic workup (Section 3.1 and Appendix B). Clinicians use ECG interpretations alongside the history and examination findings. We therefore retain the ECG conclusion as intermediate evidence and reserve discharge diagnoses for scoring. Withholding only the conclusion field would test sensitivity to reduced evidence. The performance change under that condition remains unmeasured.

Data and scoring. We documented label adjudication, confirmed that the training and evaluation sets share zero patients, and traced all 1,181 evaluation references to source entries flagged as exact ICD-10 matches (Appendices A and B). We rescored 39 archived prediction sets to test scoring sensitivity and the effect of retaining only the first diagnosis. The mean composite-score gain from adding the full agent to SFT is 73.5% on full lists and 43.7% after truncation. We also report secondary-diagnosis recovery and pair ordering on the 36 multi-reference cases (Appendix E.1).

Clinician rankings and retrieval. On 24 stratified cases, one clinician and the composite scorer agreed on 37 of the 43 pairs that both strictly ordered. Their aggregate system rankings differed (Appendix G.1). We also compared five Lingshu-7B inference and retrieval configurations on all 1,088 evaluation cases and report diagnostic performance, model calls, and token use (Appendix C, Table 6).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.