- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
“The paper presents ViMed-Struct, a Vietnamese dataset for turning clinical speech into structured medical records. Real hospital records are converted into narratives, which healthcare workers read aloud in quiet and noisy conditions, giving about 48 hours of audio over 1,475 cases. The paper benchmarks speech models and ASR-plus-LLM pipelines, scores field placement and value quality separately, and proposes a loss that weights each record field equally during fine-tuning.”
Critical strengths
The resource is real and hard to build. It is grounded in hospital records, recorded by healthcare workers rather than synthesized, and includes matched quiet and noisy recordings of the same scripts (Section 3.2). Deriving targets from existing records removes the need for manual structured annotation of audio, which is a practical recipe for other low-resource settings. Separating where information is placed (Presence F1) from what is extracted (Value Quality) is well motivated, and Figure 4 shows clearly how high ROUGE-L and BERTScore can hide a wrong field assignment. The benchmark is broad (four recognizers, four extractors, three adaptation settings, two direct models), and the reference-transcript setting (Table 5) is the right design for separating ASR errors from extraction errors.
Critical weaknesses
A record-grounded Vietnamese clinical speech resource is genuinely useful, and the field-aware evaluation is a good idea. My score is driven by three issues, listed first under Concerns. Several quantitative claims in the text don't match the paper's own tables, the evaluation setup can't yet support the reported differences, and the method contribution is small relative to ICLR's bar. If the authors correct the inconsistencies, confirm a speaker-disjoint test set, and report per-field results with uncertainty, I would raise my score to 5.
Section 4.2.2 says that replacing PhoWhisper+LoRA transcripts with reference transcripts improves all twelve extractor settings, raising macro Presence F1 by 0.033 to 0.119. Comparing Table 3 (PhoWhisper-medium+LoRA, quiet) with Table 5 gives a different picture. MedGemma ZS falls from 0.913 to 0.72 and Qwen2.5 ZS falls from 0.769 to 0.71. MedGemma FS rises by about 0.32, and Gemma ZS and LoRA rise by under 0.02. The VQ-ROUGE-L claim (0.087 to 0.259) fails in the same way, since MedGemma ZS drops from 0.686 to 0.39. Gemma-4-12B LoRA has macro Presence F1 0.93 and VQ-ROUGE-L 0.77 in Table 5, but 0.908 and 0.749 in Table 7, although both appear to be the same reference-transcript LoRA setting. In Table 3, the Whisper-large-v3 MedGemma ZS and LoRA rows are identical in all eight columns, which looks like a copy error.
The test set has 143 cases. Family history is present in 1.0% of cases (Figure 2), so the test set likely holds one or two positives, and that field's F1 is nearly noise or undefined. Yet it counts equally in macro Presence F1. Fields present in 97 to 99% of cases are close to trivial. No seeds, standard deviations or confidence intervals are reported. The paper doesn't say whether metrics are computed per case or per recording (283 test recordings). The split used to choose α in Table 6 isn't stated. If it was the test set, Table 7 is optimistic. The abstract's "up to 94%" macro Presence F1 comes from extraction on reference transcripts (Tables 5 and 7). The best end-to-end speech result is 0.917 (GPT-Audio-1.5) or 0.914 for the best cascade (Table 3). The abstract should say which setting the 94% refers to.
Questions for authors
Can you reconcile the Section 4.2.2 claims with Tables 3 and 5, and the Gemma LoRA numbers between Tables 5 and 7? Are test speakers disjoint from training speakers? If not, what is the PhoWhisper LoRA WER on held-out speakers? Which split was used to choose α, and does Macro Field Loss help when the extractor reads ASR transcripts rather than reference transcripts? What type and level of noise defines the noisy condition?