acceptodds
Withdrawn from ICLR 2027

ViMed-Struct: Structured EMR Generation from Vietnamese Clinical Speech

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Clinical documentation requires physicians to repeatedly convert information conveyed during patient encounters into structured electronic medical record (EMR) fields, creating a substantial workload. Automating this process from speech is promising, yet progress is limited by the scarcity of paired clinical audio and structured EMR annotations. This bottleneck stems not only from privacy and practical constraints on clinical audio recording, but also from the high cost of structured data annotation, particularly for low-resource languages. To address this gap, we introduce a framework for constructing aligned clinical audio - structured EMR datasets from hospital records. By deriving structured targets directly from existing EMR data and separating them from speech realization, the framework removes the need for costly manual structured annotation while enabling scalable diversification of speakers, speaking styles, and acoustic conditions. Using this framework, we build ViMed-Struct, to the best of our knowledge the first large-scale Vietnamese speech-to-structured-EMR dataset, comprising 2,572 clinical cases and 2,892 recordings totaling 47.99 hours, collected from 58 speakers under matched quiet and noisy conditions. We further establish a benchmark with a field-aware evaluation metric that measures both correct field selection and extracted value quality, providing a more faithful evaluation than conventional text-similarity metrics alone. Finally, we propose an ASR - Information Extraction framework for speech-to-structured-EMR prediction and compare it with representative existing approaches. Experiments on ViMed-Struct show that our method achieves up to 94% macro-averaged field-presence F1, demonstrating strong performance in structured extraction across diverse backbone models.

Primary area:
datasets and benchmarks
Keywords:
Medical Transcription, Speech Recognition, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XnB6OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

“The paper presents ViMed-Struct, a Vietnamese dataset for turning clinical speech into structured medical records. Real hospital records are converted into narratives, which healthcare workers read aloud in quiet and noisy conditions, giving about 48 hours of audio over 1,475 cases. The paper benchmarks speech models and ASR-plus-LLM pipelines, scores field placement and value quality separately, and proposes a loss that weights each record field equally during fine-tuning.”

Critical strengths

The resource is real and hard to build. It is grounded in hospital records, recorded by healthcare workers rather than synthesized, and includes matched quiet and noisy recordings of the same scripts (Section 3.2). Deriving targets from existing records removes the need for manual structured annotation of audio, which is a practical recipe for other low-resource settings. Separating where information is placed (Presence F1) from what is extracted (Value Quality) is well motivated, and Figure 4 shows clearly how high ROUGE-L and BERTScore can hide a wrong field assignment. The benchmark is broad (four recognizers, four extractors, three adaptation settings, two direct models), and the reference-transcript setting (Table 5) is the right design for separating ASR errors from extraction errors.

Critical weaknesses

A record-grounded Vietnamese clinical speech resource is genuinely useful, and the field-aware evaluation is a good idea. My score is driven by three issues, listed first under Concerns. Several quantitative claims in the text don't match the paper's own tables, the evaluation setup can't yet support the reported differences, and the method contribution is small relative to ICLR's bar. If the authors correct the inconsistencies, confirm a speaker-disjoint test set, and report per-field results with uncertainty, I would raise my score to 5.

Section 4.2.2 says that replacing PhoWhisper+LoRA transcripts with reference transcripts improves all twelve extractor settings, raising macro Presence F1 by 0.033 to 0.119. Comparing Table 3 (PhoWhisper-medium+LoRA, quiet) with Table 5 gives a different picture. MedGemma ZS falls from 0.913 to 0.72 and Qwen2.5 ZS falls from 0.769 to 0.71. MedGemma FS rises by about 0.32, and Gemma ZS and LoRA rise by under 0.02. The VQ-ROUGE-L claim (0.087 to 0.259) fails in the same way, since MedGemma ZS drops from 0.686 to 0.39. Gemma-4-12B LoRA has macro Presence F1 0.93 and VQ-ROUGE-L 0.77 in Table 5, but 0.908 and 0.749 in Table 7, although both appear to be the same reference-transcript LoRA setting. In Table 3, the Whisper-large-v3 MedGemma ZS and LoRA rows are identical in all eight columns, which looks like a copy error.

The test set has 143 cases. Family history is present in 1.0% of cases (Figure 2), so the test set likely holds one or two positives, and that field's F1 is nearly noise or undefined. Yet it counts equally in macro Presence F1. Fields present in 97 to 99% of cases are close to trivial. No seeds, standard deviations or confidence intervals are reported. The paper doesn't say whether metrics are computed per case or per recording (283 test recordings). The split used to choose α in Table 6 isn't stated. If it was the test set, Table 7 is optimistic. The abstract's "up to 94%" macro Presence F1 comes from extraction on reference transcripts (Tables 5 and 7). The best end-to-end speech result is 0.917 (GPT-Audio-1.5) or 0.914 for the best cascade (Table 3). The abstract should say which setting the 94% refers to.

Questions for authors

Can you reconcile the Section 4.2.2 claims with Tables 3 and 5, and the Gemma LoRA numbers between Tables 5 and 7? Are test speakers disjoint from training speakers? If not, what is the PhoWhisper LoRA WER on held-out speakers? Which split was used to choose α, and does Macro Field Loss help when the extractor reads ASR transcripts rather than reference transcripts? What type and level of noise defines the noisy condition?

Reviewer_fut7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces ViMed-Struct, a Vietnamese clinical speech-to-structured-EMR benchmark constructed from de-identified hospital records. The dataset comprises 2,572 cases, of which 1,475 have associated audio (2,892 recordings, totaling 47.99 hours). The paper evaluates direct speech-to-structure models and cascaded ASR–LLM systems, proposes separate metrics for field presence and value quality, and introduces Macro Field Loss to balance token-level supervision across structured fields.

Critical strengths
  1. The paper addresses a practically relevant problem with relatively limited existing Vietnamese clinical resources. Grounding structured targets in hospital records is a reasonable strategy for reducing annotation costs.

  2. Comprehensive experimental setup. Authors compared cascaded and direct systems under different acoustic conditions, including experiments with reference transcripts, is a potentially informative experimental design.

Critical weaknesses
  1. Limited real-world validity. The benchmark uses scripted clinical speech rather than natural clinical interactions. Section 3.2 (p. 4, lines 209–213) states: "Each verified clinical script is read aloud and recorded by an experienced healthcare worker under a controlled reading protocol." This is an important limitation of the benchmark. The speech is generated by reading narratives derived from existing structured records, rather than by recording spontaneous clinical encounters. The protocol also instructs speakers to preserve content and avoid additional filler information.

Consequently, the benchmark may underrepresent phenomena encountered in realistic clinical documentation, including spontaneous reformulation, interruptions, incomplete utterances, corrections, and dialogue-driven information discovery. The proposed benchmark appears closer to structured extraction from scripted clinical dictation than to general speech-to-EMR generation.

My suggestions. The authors should explicitly restrict their claims to this setting or provide external validation on naturally occurring clinical dictation. A description of the noise-generation protocol and its correspondence to realistic hospital acoustics is also needed.

  1. Inadequate semantic and clinical evaluation: The absence of LLM-as-a-judge is a significant limitation. Section 3.4 (p. 6, lines 269–285) defines Presence F1 using binary null/non-null decisions and Value Quality using ROUGE-L and BERTScore over populated reference free-text fields. Although these metrics provide useful diagnostic information, they are insufficient for evaluating the semantic correctness of structured clinical records. In particular, Presence F1 only evaluates whether a field is populated, irrespective of whether its content is correct. ROUGE-L measures lexical overlap, while BERTScore estimates semantic similarity through contextual embeddings. Neither directly verifies whether the generated EMR faithfully preserves the clinical information supported by the source speech. Given the availability of capable contemporary LLMs, the absence of an LLM-as-a-judge evaluation is a notable missed opportunity. Such an evaluation could assess semantic correctness at the field level rather than relying exclusively on string-overlap or embedding-similarity measures.

My suggestions. Specifically, an LLM judge could be provided with the reference clinical transcript, the canonical structured target, and the predicted structured output, and asked to assess: (1) Whether each predicted value is clinically and semantically correct. (2) Whether the predicted value is supported by the source transcript. (3) Whether important clinical information is omitted. (4) Whether information is assigned to the appropriate schema field. (5) Whether the prediction introduces unsupported clinical facts or changes clinically important details. ==> This would enable a more informative analysis of errors that may be obscured by the current evaluation protocol.

  1. Reproducibility are insufficiently established. Section 3.3 (p. 5, lines 245–256) reports a case-disjoint split of the audio dataset, with only 143 cases in the test set. The paper does not clearly establish patient-disjoint, speaker-disjoint, or hospital-disjoint evaluation. This is important because case-level separation does not necessarily prevent the same patient's records from appearing across splits, while sharing speakers or hospital-specific documentation patterns may yield overly optimistic estimates of generalization ==> Data leakage

My suggestions. The authors should clarify the split criteria and, where possible, include speaker-held-out or hospital-held-out analyses.

  1. Limited data novelty. The paper's novelty over existing clinical speech-to-structure research is unclear. Prior work (Suominen et al., 2015; 2016), already cited in Table 1, explored clinical speech recognition, structured information extraction, and field-level evaluation. ViMed-Struct largely extends this established framework to Vietnamese clinical data with a larger dataset and broader schema.

  2. Limited result novelty. The three research questions largely revisit well-established findings: (RQ1) comparing direct speech-to-structure models with ASR–LLM pipelines, (RQ2) decomposing ASR and downstream extraction errors, and (RQ3) examining field-level performance imbalance. These issues have been explored in prior clinical speech-to-structure studies, including Corbeil et al. (2025) and Su et al. (2026), or even in many related works. The main potential novelty therefore lies in the Vietnamese dataset, rather than in new methodology or scientific insights.

  3. Limited benchmark difficulty and potential saturation. Table 3 shows that GPT-Audio-1.5 already achieves 0.917/0.918 macro Presence F1 on quiet/noisy speech in a zero-shot setting, suggesting limited headroom for stronger frontier models. Moreover, the dataset consists of structured EMRs converted into scripts and read aloud under controlled conditions, potentially making the task substantially easier than real-world clinical documentation. Unlike datasets containing natural, spontaneous medical speech, such as MultiMed ASR dataset https://arxiv.org/abs/2409.14074, this benchmark lacks realistic speech variability and conversational complexity. I am concerned that its primary metric could be rapidly saturated by stronger models.

My suggestions. The authors should demonstrate greater benchmark difficulty and long-term discriminative value through more challenging, spontaneous clinical speech and clinically grounded semantic evaluation.

Questions for authors

I appreciate the authors' efforts to develop a Vietnamese clinical speech dataset, which could be a valuable contribution to the Vietnamese NLP and healthcare communities. However, I believe the paper's main contribution is in dataset construction and benchmarking rather than methodological novelty. I encourage the authors to address the evaluation and generalization concerns and consider submitting a revised version to a more resource-oriented venue, such as LREC, where this contribution may be better appreciated.

My intention is to provide constructive feedback rather than simply advocate rejection. I sincerely hope the authors continue improving this work and ultimately succeed in publishing it.

My criticisms and initial hypotheses were based on my own assessment of the paper. I used AI to help investigate potential technical issues, cross-check reported results, and polish my review. I remain responsible for verifying the accuracy of these observations and for my final assessment.

Wish all luck to authors!

Reviewer_8HXXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ViMed-Struct, a Vietnamese speech-to-structured-EMR dataset constructed from hospital records, with 47.99 hours of recordings from 58 speakers. It establishes a benchmark covering direct speech models and ASR-based pipelines, evaluates field presence and value quality separately, and reports up to 94% macro field-presence F1 for structured extraction.

Critical strengths
  1. The dataset provides a useful resource for Vietnamese speech-to-structured-EMR research. Deriving targets from existing hospital records reduces manual structured annotation, while verified scripts connect the recorded speech to the target clinical content.
  2. The benchmark supports controlled analysis of recognition and extraction errors. It includes quiet/noisy recordings of the same content, direct and cascaded systems, and extraction from reference transcripts, with case-disjoint data splits.
Critical weaknesses
  1. The claimed separation of ASR and extraction errors is not consistently supported by the reported results. Section 4.2.2 states that reference transcripts improve all twelve configurations, but zero-shot MedGemma scores 0.913 in macro Presence F1 with adapted ASR in Table 3, versus 0.72 with reference transcripts in Table 5. This discrepancy needs clarification before the effect of recognition errors can be assessed reliably.
  2. The benchmark's representativeness of clinical speech remains insufficiently established. All evaluated recordings are readings of scripts derived from the target EMRs. Although the scripts are rephrased, reordered, and manually verified, both training and test data follow the same construction process. Case-disjoint splits therefore do not establish whether extraction performance transfers beyond this process. This limits the evidence that the benchmark captures the difficulty of structuring independently formulated clinical speech, rather than recovering records from their edited verbalizations.
Questions for authors

To what extent does performance on record-derived, read-aloud scripts reflect structured extraction from spontaneous clinical speech, and what evidence supports this connection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.