acceptodds
Under review as a conference paper at ICLR 2027

SLAM: A Self-Hostable Multi-Backend Reconciler for Real-Time Medical ASR

Abstract

Medical automatic speech recognition (ASR) must recover safety-critical content while also applying formatting and editing operations to a clinical document. We study multi-backend ASR reconciliation, in which an LLM combines hypotheses from heterogeneous ASR systems into a single corrected and formatted output. We focus on a real-time per-chunk configuration that avoids the additional cost and state required by revisable rolling-window reconciliation. To make this configuration practical to deploy, we investigate replacing its proprietary frontier-model reconciler with a self-hostable open-weight student. We train students from the teacher's reconciliations using real clinician speech, synthetic speech, and their combination. Synthetic examples are language model generated, rendered with text-to-speech, and acoustically augmented to induce disagreement between the production ASR backends. We evaluate transcription and content fidelity on 226 held-out real clinical encounters and more complex formatting behavior on a separate 61-case targeted suite. Our best 27B student approaches the teacher, with no statistically significant difference in either word error rate (WER) or keyword error rate (KER). Synthetic supervision is competitive with real supervision at the evaluated training budget, while blended supervision matches or exceeds real-only supervision on the behavioral suite at every model size. Acoustic augmentation improves KER at every evaluated model size, with the largest observed gain for the smallest student. Self-hosting reduces measured reconciler cost by approximately x in our deployment comparison, and a 35B mixture-of-experts student reduces median reconciliation latency from 1,100,ms to 466,ms. These results show that the frontier-model reconciler in real-time multi-backend clinical ASR can be replaced by a faster, lower-cost, self-hostable model, with much of the required supervision generated synthetically.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
clinical ASR, synthetic data, small language models, fine-tuning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_KHqPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies a clinical ASR pipeline where ASR backends independently transcribe speech chunks, and an LLM reconciles their outputs using clinical and document context. The proposed training pipeline uses synthetic clinical scripts, text-to-speech, acoustic perturbations, production ASR hypotheses, and teacher-generated targets. Eval is conducted on 226 synthetic clinical encounters and a separate 61-case behavioral suite.

Critical strengths
  1. Medical ASR is a practical problem because general-domain ASR might not work very well in clinical settings where clinical terms might give some more difficulties for the joint Speech-Language models to transcribe well

  2. Self-hostable and real-time ASR is a interesting approach because in clinical settings users might not want their datasets uploaded online which violates patients-doctors privacy

Critical weaknesses

I suspect this paper is an AI slop, based on some main points here:

  1. Related works are not sufficiently included. The authors conduct study on medical ASR but there is no work on medical ASR cited. I asked ChatGPT and it easily gave me many publicly available datasets in just 1 min. For example:

Publicly available medical ASR datasets

Real-time medical ASR setting

  1. The synthetic generation of the dataset is very naive and thus makes almost no real-world relevant. The authors put some dictation script and used synthetic TTS to generate synthetic audio files, and then augmented with noise and pertubations. Such medical ASR dataset is very far from real-world settings where noise, speaker speed and vocab is very diverse and varying (see the real-world datasets by VietMed and MultiMed).

  2. The experimental results are not convincing if it is implemented in real-world hospitals. As I can see in main Table 2, WER is below 0.1xxx. Such perfect results are generated simply based on the synthetic dataset which is simply a reverse-engineered of the same frontier models. So I'm not surprised about such low error rates reported in the paper.

Questions for authors

N/A

2 comments

Question 1: Related work

Thank you for these suggestions. VietMed and MultiMed focus on multilingual medical ASR datasets and recognition models, while the third work addresses clinical speech summarization. Our paper instead studies reconciliation of multiple ASR outputs and the distillation of a teacher model into a student reconciler.

We focus our related work discussion on multi-backend ASR reconciliation and generative ASR error correction, which are most directly relevant to our research question (Section 2.1). Additionally, we do not focus on multilingual ASR in this paper, but agree that it would be an interesting direction for future work, as noted in Section 7 (Discussion).

Question 2: Real-world relevance of synthetic data

We agree that TTS cannot fully reproduce the acoustic diversity of real clinical speech. However, we use synthetic audio for training supervision, not evaluation (Sections 4–5). We also apply acoustic augmentation to introduce more diverse ASR disagreements, including noise, reverberation, and speed perturbations (Section 4; Appendix D), and evaluate its contribution through a controlled ablation (Section 6.4).

Importantly, all models are evaluated on the same 226 held-out real clinical encounters, spanning 15 clinicians and 28 specialties, with no clinician overlap between training and evaluation (Section 5, Datasets; Table 2).

We acknowledge that synthetic speech does not fully capture real-world acoustic variability, but our evaluation on held-out real clinical speech directly measures how well synthetic supervision transfers to the target setting.

Question 3: Real-world validity of the experimental results

Thank you for raising this concern. To clarify, all results in Table 2 are evaluated on real clinical speech. ClinStandard comprises 226 held-out real clinical encounters from 15 clinicians across 28 specialties, with independently annotated references and no clinician overlap with training (Section 5, Datasets; Table 2).

We have clarified this distinction in the manuscript and specified that WER values are proportions, not percentages (e.g., 0.1336 corresponds to 13.36% WER; Section 5, Metrics; Table 2 caption).

While these results demonstrate performance on held-out real clinical speech, we do not claim generalization to all hospitals or acoustic environments (Section 7, Limitations).

Thank you to the authors for taking the time to prepare a thoughtful rebuttal!

I appreciate the authors' efforts, but I feel that only W3 adequately addresses my concerns.

Regarding W1, I believe that when working on a domain X, it is important to discuss and cite relevant prior work in that domain, so that readers can understand the current state of the art and the research directions being explored by the community. Furthermore, when proposing a new method Y, I would expect evaluations on both the newly introduced benchmark and existing benchmarks in X to demonstrate the generalizability of the proposed method (VietMed and MultiMed in your case).

For these reasons, I will respectfully maintain my original score. That said, I hope the authors will not be discouraged by this assessment. I appreciate the effort that went into this work and encourage the authors to continue developing and improving it. I wish them all the best with their future research!

Reviewer_WuboOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose SLAM, which reconciles transcripts from two ASRs using a LLM, aiming to replace a proprietary reconciler with a smaller self-hostable one. They distil into four Qwen3.5 model sizes from real clinician speech, TTS-generated synthetic speech, and a blend, using sequence-level distillation and LoRA.

Critical strengths
  • I like that the authors evaluate how the model edits clinical notes, rather than only looking at transcription accuracy. The ClinBehavioral benchmark is a useful contribution, since a low WER doesn't necessarily mean the model handles clinical documentation correctly. They showed why evaluating the actual behavior is important.
  • The evaluation using real clinical encounters across several specialties is another strong point, especially compared with studies relying on read or simulated speech. I also find the acoustic augmentation approach interesting, since it creates disagreements between transcription systems that are otherwise difficult to collect.
Critical weaknesses
  • I'm concerned that KER only measures missing keywords but doesn't penalize incorrect information being added. E.g., a wrong medication or doses could go unnoticed by this metric, even though such errors are important in clinical practice.
  • Although WER accounts for insertions, it doesn't distinguish between minor transcription mistakes and probably also not clinical errors. I would suggest adding a clinical entity-level evaluation that measures precision and recall for medications, doses, and numerical values, including how often the model introduces incorrect information.
  • The comparison in Tab.6 is described as size-matched, but the dataset has almost twice as many samples as either individual dataset. This makes it difficult to tell whether the improvements come from combining real and synthetic data or simply using more training examples. An ablation comparison using the same number of training samples would help clarify this.
  • I'm also not fully convinced whether blending consistently improves performance. For the 27B model, real-only training actually gives slightly better WER/KER, with the same number of behavioral passes. Other model sizes show some benefits, but I think the conclusions should reflect these differences rather than suggesting that blending is always better. And also limitation should be stated regarding this.
Questions for authors
  1. How often does the model introduce incorrect clinical information, such as medications, doses, or numerical values? Can this be evaluated using the existing predictions?
  2. Are the data splits separated by clinician or by encounter? Does the 27B result still hold when accounting for multiple encounters from the same clinician?
  3. Do you have results comparing real, synthetic, and blended data using the same training-set size? How much of the improvement comes from having more data?
  4. What is the actual cost per session for the 27b+ models? Do the reported cost savings also apply to these models?
  5. What is the variability in KER across training seeds? Are the small improvements consistent across runs?
3 comments

Question 1: Incorrect clinical information

Thank you for highlighting this limitation. We adopted KER based on prior ASR literature (Park et al., 2008; Le-Duc et al., 2025) to emphasize the preservation of clinically relevant terminology. However, we agree that our reference-focused formulation does not penalize unsupported clinical insertions.

To address your concern, we added clinical entity precision and recall, motivated by prior work on factual precision and recall in medical documentation (Ben Abacha et al., 2023). Unlike KER, entity precision penalizes medications and numerical values appearing in the prediction but absent from the reference.

On 226 real ClinStandard encounters, 27B Real achieves medication precision of 0.976 versus 0.969 for the teacher, with slightly lower recall (0.955 versus 0.969). Numerical precision is similarly close (0.977 versus 0.979). These metrics do not assess whether medications are associated with the correct doses. Our additional error analysis also identified 37 potential medication insertions that have not been adjudicated against source audio.

As these additional metrics do not materially change our conclusions, we report the full methodology and results in Appendix I (Table 17).

Question 2: Clinician-level independence

Thank you for raising this important question about clinician-level independence.

ClinStandard comprises 226 real clinical encounters from 15 clinicians spanning 28 specialties, with no overlap between training and evaluation clinicians. We have clarified this distinction in the revised Datasets subsection (Section 5).

To account for multiple encounters from the same clinician, we additionally performed clinician-clustered bootstrapping. We find that the 27B Real student remains statistically comparable to the teacher on WER, within our predefined 10% relative margin. The WER parity criterion holds under both encounter-level and clinician-clustered resampling, while the KER criterion holds only under encounter-level resampling.

We have added the full statistical analysis to Appendix J (Table 18) and updated our conclusions accordingly.

Question 3: Size-matched supervision comparison

Thank you for identifying this confound. Our motivation for blending was to observe the effects of combining real and synthetic data to play to each of their strengths.

As you correctly noted, our original comparison conflated data composition with training volume. To address this, we performed an additional ablation with a size-matched Blend (2,866 examples).

At comparable training sizes, Blend still helps student models pass more ClinBehavioral test cases across all model sizes, although these differences remain statistically uncertain. This supports our hypothesis that blending may benefit targeted reconciliation behaviors. A larger blended dataset further improves WER relative to size-matched Blend, highlighting the importance of training volume and the value of synthetic data as a lower-cost source of additional supervision.

In response to your feedback, we have incorporated the controlled comparison into Table 2 (Section 6.1), added the statistical analysis to Appendix H (Table 16), and revised our conclusions to distinguish data diversity from training volume.

Question 4: Cost of larger models

Thank you for requesting this clarification. We have updated Table 1 (Section 3.2) to reflect the serving costs of all student models. The 27B model runs on two H100s, with an estimated reconciler cost of 1.728 cents per session, compared with 4.306 cents for the frontier model (approximately 2.5× cheaper). The same estimated cost applies to the 35B-A3B model.

Please also note that it is possible to run the 27B model on a single H100 with various trade-offs, which could further reduce serving costs. Our reported comparison, however, uses the two-H100 configuration.

Question 5: Training-seed variability

Thank you for raising this concern. We initially trained three seeds for Synthetic at each model size. At the main training budget, KER variability was low (standard deviation ≤0.003\leq 0.003). Given this limited variance and the computational cost of additional runs, we proceeded with single-seed evaluations for the remaining configurations.

We acknowledge that seed variability may differ across supervision sources and training budgets, particularly where observed performance differences are small.

  1. My concern here are mostly resolved. However, I want to point out that precision alone does not establish clinical correctness. A model might transcribe the correct medication and numerical value but associate them incorrectly. From those 37 potential medication insertions, were these hallucinations or errors in the reference?
  2. Thank you so much for the confirmation that no clinician overlap and that you performed clustered bootstrapping. However, the 27B model meets their WER parity criterion under clinician clustering, but not the KER criterion. 3 and 4 Fully resolved.
  3. The multiple seed are only evaluated for the syntethic, what about the other models with real/blend?

(1) Thank you for raising this point. Without clinician verification, we assessed the 37 potential medication insertions using two independent LLM judges (Claude Opus 5.5 and GPT-5.6) and a fixed rubric (35/37 agreement, κ = 0.88). Neither identified a reference error. They agreed on 25 drug substitutions, 7 fabrications, and 3 equivalent or partial names. In 11 cases, a correct dose or frequency was assigned to the wrong drug, an error KER does not fully capture.

We also reviewed the teacher's 12 insertions, finding 3 reference errors and 2 potentially high-harm fabrications. These findings suggest the issue is not unique to distilled students, although the classifications have not been clinically validated.

(2) You're correct. The 27B student meets our predefined WER parity criterion under clinician clustering, but not the KER criterion. We've clarified this distinction and limit the KER parity claim to encounter-level analysis.

(3) Thank you for pointing this out. We ran multiple seeds only for Synthetic and did not extend the analysis to Real or Blend because the initial results showed limited variability. We acknowledge that this may not generalize across training distributions.

Reviewer_aA3vOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates replacing a proprietary frontier LLM-based reconciler in a real-time medical ASR system with self-hostable open-weight student models. The authors introduce a synthetic supervision pipeline and train Qwen-based student reconcilers using synthetic, real, and blended supervision. Experiments show that the student models achieve competitive reconciliation performance while reducing inference cost and latency.

Critical strengths
  1. The paper addresses an important practical bottleneck in medical ASR deployment. Reliance on proprietary frontier LLMs introduces substantial cost, latency, and deployment constraints in medical applications. By distilling reconciliation capabilities into self-hostable open-weight student models, the paper offers a practical approach to improving deployment efficiency in healthcare settings.
  2. The synthetic supervision pipeline is a meaningful contribution. The proposed pipeline combines synthetic speech generation, acoustic augmentation, and ASR ensemble outputs to induce realistic recognition errors and disagreements. Using teacher-generated reconciliations as supervision provides a practical alternative to relying solely on real clinical speech data.
Critical weaknesses

Weakness 1: ClinBehavioral contains only 61 cases, with some behavioral categories having very few examples . The absence of category-level success rates and uncertainty estimates makes it difficult to assess the reliability of the reported behavioral improvements.

Weakness 2: The distillation pipeline relies on a proprietary teacher model, but key details such as its exact version, prompting strategy, and inference configuration are not sufficiently documented. This limits the reproducibility of the training process.

Weakness 3: The student reconciler is trained and evaluated using a fixed ASR ensemble, which may lead to learning backend-specific error patterns. Its ability to generalize to different ASR backends or changing error distributions remains unclear.

Weakness 4: Table 2 reports median (p50) latency, demonstrating the efficiency of the student models, but provides limited characterization of tail latency. Additional p95/p99 measurements across different input lengths would strengthen the evidence for real-time deployment robustness.

Weakness 5: The evaluation mainly focuses on internal baselines, with limited direct comparison to existing ASR error correction approaches.

Questions for authors

Question 1: Could the authors provide category-level success rates and uncertainty estimates for ClinBehavioral, such as confidence intervals or paired bootstrap analysis, to better assess the reliability of the reported improvements?

Question 2: Could the authors provide further details about the proprietary teacher model, including its version, prompting strategy, and inference configuration, to improve reproducibility?

Question 3: Could the authors further evaluate how well the student reconciler generalizes when the upstream ASR backends or their error distributions change?

Question 4: Could the authors report p95/p99 latency measurements for the student models and teacher, ideally across different input lengths, to better characterize real-time deployment performance?

Question 5: Could the authors consider including comparisons with representative existing approaches to better contextualize the proposed method's performance?

2 comments

We thank the reviewer for their thoughtful questions and constructive suggestions. We address each question below.

Question 1: ClinBehavioral category-level results and uncertainty estimates

Thank you for requesting a more granular evaluation. We added category-level ClinBehavioral results and statistical analysis for the 61-case suite.

Fine-tuning substantially improves behavioral performance over the base models at 4B, 9B, and 35B across all supervision sources. Full Blend also matches or exceeds the pass counts of Real and Synthetic supervision across all model sizes, although individual comparisons are often inconclusive due to small sample sizes.

In response to your feedback, we added the category-level results and significance tests to Appendix G (Table 15), with the size-matched analysis in Appendix H, while acknowledging the limited statistical power within individual categories.

Question 2: Proprietary teacher configuration

Thank you for identifying this gap. We have expanded the appendix to report the teacher's configuration. Notably, the teacher model is gpt-5.6-terra, accessed through the Chat Completions API with reasoning effort set to none, temperature 0.0, a maximum completion length of 16,384 tokens, and outputs constrained to a strict JSON schema. The prompting strategy is described at a high level in Section 3.1, where the reconciler uses both ASR hypotheses, existing document context, and structured patient information to produce a corrected and formatted output. The remaining inference details are provided in the newly added Appendix F.4 (Teacher Configuration).

Question 3: Generalization across ASR backends

Thank you for raising this important point. The upstream ASR backends are already part of our production pipeline and are therefore held fixed throughout our experiments. Our focus is on improving the downstream reconciliation component within this controlled setting. We agree that evaluating robustness to changes in ASR backends or error distributions would be valuable. We acknowledge this limitation in Section 7 (Discussion) and identify cross-backend generalization as an important direction for future work.

Question 4: Tail latency measurements

Thank you for suggesting the inclusion of tail latencies. Initially, we included p50 for compactness because the latency profiles followed similar patterns. Nonetheless, we have now added p95 and p99 per-call reconciliation latency measurements in Table 2 (Section 6.1) for completeness.

For example, the 35B-A3B student achieves p50/p95/p99 latencies of 466/562/571 ms, compared with 1,100/1,778/2,502 ms for the teacher. This represents approximately 2.4× lower median latency and 4.4× lower p99 latency.

Please note that the teacher model is also using a priority tier which is more expensive and performant than the default tier. Another interesting observation is smaller increase in tail latencies as compared to the median latencies for the self-hosted models as compared to the teacher model.

Question 5: Comparisons with existing approaches

Thank you for this suggestion. We agree that comparisons with established ASR error-correction methods would provide useful additional context.

Our evaluation focuses on the frontier reconciler and untuned student models because our primary question is whether a smaller model can reproduce the quality and document-editing behavior of a production reconciliation system. Unlike conventional ASR hypothesis combination, SLAM must reconcile competing transcripts while handling cursor placement, selection replacement, and multi-chunk document assembly.

We discuss representative existing approaches in Section 2.1, including ROVER (Fiscus, 1997), generative ASR error correction (Chen et al., 2023), and confusion-network-based reconciliation (Ma et al., 2025). Our preliminary work also compared multi-backend reconciliation with single-backend ASR followed by LLM post-processing, finding an approximately 40% reduction in normalized KER, as summarized in Section 3.2 and Appendix A.

We recognize, however, that these task differences do not eliminate the value of direct comparisons with existing approaches. We have clarified the scope of our evaluation and acknowledge the absence of external error-correction baselines as a limitation. Such comparisons remain an important direction for future work.

We thank the reviewer again for their constructive feedback.

Thank you for the clarifications and the additional analyses. The addition results improve the presentation and make the evaluation more informative.

However, my main concerns remain only partially addressed. In particular, the ClinBehavioral suite is still very small, and the added analysis does not fully resolve the reliability concern for behavior-level claims. The generalization of the student reconciler beyond the fixed ASR ensemble remains unevaluated, and the paper still lacks direct comparisons with representative existing ASR error-correction or reconciliation baselines. These issues limit how broadly the reported results can be interpreted.

Therefore, I would keep my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.