acceptodds
Under review as a conference paper at ICLR 2027

MedSonor: Learning to Use Clinical Audio in Medical LLMs

Abstract

Clinical audio conveys medical information that transcripts may not preserve, but using it in medical responses requires interpreting it alongside clinical requests and dialogue context. Existing medical audio supervision largely targets classification or recognition, while general audio-language models lack specialized medical knowledge and direct feedback for open-ended responses. We present MedSonor, a medical dialogue model integrating medical knowledge, spoken dialogue, and clinical audio understanding. We pair authentic medical audio with clinical requests and combine these audio-integrated dialogues with collected medical audio and synthesized spoken dialogues for supervised fine-tuning, followed by two stages of rubric-guided reinforcement learning. We introduce MedVoice-Dial and MedVoiceAID to evaluate responses within supplied multi-turn medical contexts and to requests involving clinical audio, respectively. Across seven medical benchmarks, MedSonor outperforms the evaluated general-purpose open-weight audio-language baselines and ranks highest in aggregate physician preference evaluation, while broadly retaining general speech capabilities.

Primary area:
reinforcement learning
Keywords:
Medical Speech Models, Multimodal Learning, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_GjwSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MedSonor adapts Qwen2.5-Omni-7B for medical audio dialogue through supervised fine-tuning and two stages of rubric-guided RL. The training data combine medical recordings, synthesized spoken consultations, and clinical requests paired with recordings. The paper also introduces MedVoice-Dial and MedVoice-AID. MedSonor outperforms the evaluated general-purpose open-weight baselines on seven medical response benchmarks and receives the highest aggregate physician preference score on 30 cases. Ablations, general speech evaluations, and a noise experiment provide additional analysis of the training pipeline.

Critical strengths
  1. The paper addresses a useful problem: recognizing a clinical sound is only part of answering a medical question about it. The combination of recordings, spoken dialogue, and audio-integrated requests in Section 3.2 is well motivated and covers different aspects of this task.

  2. The source ablations help explain where the gains come from. In particular, removing audio-integrated SFT lowers MedVoice-AID from 0.7900 to 0.5957 even though both RL stages are retained (Table 19). This is a substantial difference and supports the value of the proposed data construction.

  3. I appreciate the effort to evaluate beyond the training rubric. The blinded physician study and the separate audio-capable pairwise evaluator provide useful checks on the automated scores (Figures 3–4 and Appendix F). Results on adapted public datasets also suggest that the improvements are not confined to the two new benchmarks.

Critical weaknesses
  1. The train/test separation of MedVoice-AID and MedVoice-Dial is not clear enough. Appendix D.1 describes disjoint splits for the local judges, but this does not establish that the policy's evaluation recordings and conversations are disjoint from all SFT and RL sources. Since the same recording can produce several scenario variants, splitting generated examples is not sufficient. Please give the test sizes, grouping rules, and overlap counts at the recording, patient/speaker where available, and conversation level. This is particularly important for MedVoice-AID, where the largest gain is reported.

  2. I am not yet convinced that the improvement measures better use of clinical acoustic evidence. GPT-5.4 generates the cases, references, and rubric supervision and also scores the main medical evaluations; the local audio judge receives reference text rather than the waveform (Appendices D.1 and G). A useful control would keep the spoken request fixed and remove or replace the clinical recording, then check whether the answer changes appropriately. Without this, it is hard to separate better acoustic interpretation from better use of contextual cues and the expected answer format.

  3. The contribution of the severity caps is not established by the current ablations. SFT alone already scores 0.7666 on MedVoice-AID, compared with 0.7900 for the full pipeline (Table 19), and there is no matched comparison with the caps disabled. Since avoiding serious medical errors motivates this part of the method, I would like to see its effect on independently assessed major/critical-error rates, together with uncertainty on the SFT-to-RL improvement.

  4. Some evaluation claims need more qualification. The physician study covers only 30 cases, and the direct preference credit against MiniCPM-o is 53.4%, so the highest aggregate rank should not be read as a clear advantage over every comparator. Also, Section 4.4 uses a 100-source subset for the noise experiment, while Table 6's clean scores exactly repeat Table 3. Please clarify whether the clean and noisy scores use the same examples; otherwise, their difference is not a matched estimate of noise sensitivity.

Questions for authors

See weaknesses 1–4.

7 comments
AuthorsOpenReview ↗

Response to Reviewer GjwS (Part 1)

We thank the reviewer for the careful assessment and constructive suggestions. We address each concern below.

W1. Evaluation sizes, grouping, and train/test separation

We appreciate the question about grouping and training overlap. We agree that the grouping unit and the limits of what the identifiers can establish should both be clear.

We checked the policy evaluation sets against the available training identifiers. For AID versus RL2 training, we found no overlap in recording families or source PCM. For SFT, we checked for exact matches in rendered audio paths.

W1.1. Evaluation sizes and independent units

Thank you for distinguishing the number of evaluation examples from the number of independent units. The latter matters for estimating uncertainty.

  • Evaluation sizes. MedVoice-AID comprises 250 requests built from 98 distinct source recordings. MedVoice-Dial comprises 250 synthetic conversations.
  • Resampling units. For the confidence intervals reported in Section W3.2, we use the source recording and the conversation, respectively, as the resampling units for these two benchmarks. Scenario variants derived from the same AID recording are grouped together during resampling.
UnitMedVoice-AIDMedVoice-Dial
Evaluated examples250250
Source recordings98 source-audio families and 98 distinct source PCM hashesSynthetic speech
Audio files supplied250 composed request-plus-recording inputs623 distinct user-audio files including history
Conversations / requests250 distinct conversation IDs representing single-turn requests250 distinct synthetic conversations

The following table lists the available identity information for sources accounting for 71 of the 98 AID recordings. Identifiers are defined within each source and should not be interpreted as globally unique patient or speaker identifiers.

Original sourceRecordings in AIDIdentity information recovered
ICBHI 20171311 patient IDs, matched to original demographic metadata
CirCor DigiScope1313 patient IDs, matched to the original patient table
Coswara1414 participant IDs, matched to original daily CSVs
SPRSound1311 recordings mapped to 11 patient IDs, while 2 remain unmapped
TORGO136 speaker IDs, traced to original speaker directories
VOICED55 record-associated IDs with original patient information files
AuthorsOpenReview ↗

Response to Reviewer GjwS (Part 2)

W1.2. Separation of policy training, judge training, and evaluation

We welcome the request to clarify data separation across stages, particularly the distinction between documented construction rules and checks we could reproduce.

The policy follows Qwen2.5-Omni-7B → medical SFT → dialogue RL (RL1) → audio-integrated RL (RL2) → benchmark inference. The two local reward judges are trained separately using GPT-5.4 criterion-level labels and provide rewards during their respective RL stages.

Stage / branchData and supervisionWhat is trained or evaluated
Medical SFT18,657 training examples: collected medical audio, synthesized spoken dialogues, and audio-integrated requests. Target responses provide supervisionAdapt the backbone to medical content and response requirements
Dialogue-judge trainingGPT-5.4 labels individual responses paired with criteria. Responses include GPT-4.1, GPT-4o-mini, and GPT-4.1-mini. The training set contains 112,514 pairsTrain a local Qwen3-4B judge using LoRA
RL16,927 dialogue examples from 1,941 conversations, each with dialogue history, current request, and associated rubricsStarting from SFT, sample response groups and update the policy with DAPO
Audio-judge trainingCurrent-policy responses plus dialogue-training replay with GPT-5.4 item labels across 22,872 training pairsTrain a separate local Qwen3-4B audio-integrated judge using LoRA
RL24,546 audio-integrated requests from 600 recording families, each combining a spoken request with a clinical recordingContinue from RL1 using DAPO and the audio-integrated judge
Benchmark evaluationAID: 250 examples, Dial: 250 examplesGenerate answers from saved policy checkpoints and score them with the external benchmark rubric scorer

The following table distinguishes the documented data-construction rules from the overlap checks supported by the available identifiers.

BoundaryConstruction rule / reproducible evidence
Dialogue judge train vs. held-outAppendix D.1 specifies disjoint conversations. Inspected files contain 1,950 training and 50 held-out conversation IDs
Audio judge train vs. held-outAppendix D.1 specifies source-family-disjoint splits and replay restricted to dialogue training data
RL1 train vs. validation1,941 vs. 56 conversations, with zero conversation-ID overlap
AID/Dial vs. SFTZero exact rendered-audio-path overlap with SFT train/validation
AID/Dial vs. RL1 and inspected judge splitsZero available sample/conversation-ID overlap with RL1 and zero conversation-ID overlap with judge splits
AID vs. RL2 trainingZero sample-ID, conversation-ID, recording-family, and source-PCM overlap
Dial vs. RL2 trainingZero sample-ID/conversation-ID overlap
AuthorsOpenReview ↗

Response to Reviewer GjwS (Part 3)

W2. Fixed-request controls for clinical acoustic evidence

We appreciate the reviewer’s question about changes to the clinical recording. Keeping the spoken request constant allows us to examine whether the answer responds to those changes.

We held the spoken request fixed and either silenced the clinical recording or replaced it with a same-organ recording carrying the opposite normal/abnormal label. All outcome values are percentages.

DesignSilenceMismatched replacement
Clinical recordingSilenced while retaining the spoken requestReplaced by a same-organ recording from a different identifiable subject with the opposite label
Evaluation cohort100 sources103 paired sources from a separate expanded cohort
Policy inputNo reference answer or rubricNo reference answer or rubric
AssessmentContext-restored GPT-5.4 reviewContext-restored GPT-5.4 review

Table 1. Silence condition. The clinical recording is silenced while the spoken request is retained.

EndpointSFTFull
Acknowledges that audio evidence is insufficient for the task conclusion40.0%73.0%
Provides task-specific verification steps65.0%79.0%
Meets the complete cautious-response criterion12.0%10.0%

Under silence, Full more often acknowledged insufficient evidence (73% vs. 40% for SFT) and suggested task-specific verification (79% vs. 65%). The complete cautious-response criterion was met in 10% of Full responses and 12% of SFT responses.

EndpointOperational definition
Insufficient task evidenceExplicitly states that the recording cannot establish the requested acoustic finding or task conclusion
Task-specific verificationAn actionable step: repeat recording/auscultation, relevant vital-sign checks, or a specified examination. Generic advice to seek care does not qualify
Complete cautious responseAdditionally withholds the audio-dependent conclusion and responds to the request, without unsupported acoustic assertions or a generic refusal

Table 2. Mismatched-recording replacement. Clinical audio remains available because only the recording is replaced, with the request unchanged.

EndpointSFTFull
Acknowledges that the recording alone is insufficient for the task conclusion55.3%83.5%
Provides task-specific verification steps62.1%78.6%
Makes the task decision conditional on further verification31.1%37.9%
Combines evidence-boundary recognition, verification, and a response to the request35.0%68.0%
Strict label accuracy for the current replacement recording26.2%36.9%

With the spoken request held fixed, strict label accuracy for the replacement recording was 36.9% for Full and 26.2% for SFT. Full also more often acknowledged the limits of the evidence, proposed verification, and addressed the request (68.0% vs. 35.0%).

W3. Severity caps, SFT-to-RL uncertainty, and severe errors

Thank you for asking about both stagewise gains and severe errors. We agree that the contribution of training and the safety of the resulting responses deserve separate examination.

W3.1. Training stages and stagewise confidence intervals

We appreciate the request for a stage-by-stage account. Paired differences between successive stages show the size and uncertainty of each change.

The following tables report scores at each training stage and differences between consecutive stages. Intervals in brackets are 95% confidence intervals. In W3.1 and W3.2, differences were computed before rounding.

DatasetnBackbone (Fig. 5)SFT (Fig. 5)RL1Full (Fig. 5)
MedVoice-AID2500.33410.76660.72900.7900
MedVoice-Dial2500.32370.42570.48230.5225
ACI-Bench5000.20020.21680.18400.2263
HealthDial2500.44890.41060.45970.5022
ThReadMed-QA2500.18000.20340.24670.3403
DatasetRL1 − SFTFull − RL1
AID-0.0376 [-0.0727, -0.0031]+0.0610 [+0.0348, +0.0871]
Dial+0.0566 [+0.0164, +0.0965]+0.0402 [+0.0024, +0.0786]
ACI-0.0329 [-0.0639, -0.0019]+0.0423 [+0.0186, +0.0662]
HealthDial+0.0490 [+0.0167, +0.0806]+0.0425 [+0.0162, +0.0688]
ThReadMed+0.0433 [+0.0011, +0.0850]+0.0936 [+0.0589, +0.1307]
  • Dialogue RL. RL1 scored higher than SFT on MedVoice-Dial, HealthDial, and ThReadMed-QA, and lower on MedVoice-AID and ACI-Bench.
  • Full policy. Full scored higher than RL1 on all five tasks, with the largest difference on ThReadMed-QA (+0.0936), and had the highest point estimate among the four stages on each task.
Reviewer_D9wVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents MedSonor, a medical audio dialogue model adapted from Qwen2.5-Omni-7B. The objective is to connect clinical acoustic evidence with medical requests and dialogue context, rather than treating medical audio understanding solely as classification or recognition.

The training pipeline combines supervised fine-tuning on collected medical audio, synthesized spoken dialogues, and audio-integrated dialogues, followed by two stages of rubric-guided reinforcement learning using DAPO. Local item-level judges assess instancespecific requirements and prohibited errors, with severity-based reward caps intended to prevent detected major or critical violations from receiving high rewards.

The authors introduce MedVoice-Dial and MedVoice-AID and evaluate MedSonor on seven medical benchmarks. The model ranks first among the evaluated general-purpose open-weight audio-language systems and achieves the highest aggregate preference ranking in a physician evaluation involving 30 cases. Additional experiments examine training-source contributions, general speech capability retention, and robustness to background noise.

Critical strengths
  1. The paper addresses a valuable distinction between recognizing a clinical sound and using that sound appropriately when answering a medical request. Its complementary supervision sources are well aligned with this objective.

  2. The evaluation covers seven medical benchmarks, general speech tasks, and background-noise robustness. The authors also conduct a physician preference study and an automated pairwise evaluation using a different model, with response-order swapping in the latter.

Critical weaknesses
  1. The experiments do not sufficiently establish that improvements depend on clinical acoustic evidence Audio-integrated requests, reference responses, and rubrics are generated by GPT-5.4 using source-audio descriptions and annotations (Section 3.2). This is a reasonable construction strategy, but it creates a potential shortcut: request wording, scenario information, or user roles may make parts of the expected answer predictable without correctly interpreting the recording. Removing a training source does not establish that the resulting model relies on the appropriate acoustic evidence at evaluation time.

  2. The specific benefits of sequential RL and severity-based reward caps are insufficiently validated. On MedVoice-AID, the backbone, SFT-only model, and full model score 0.3341, 0.7666, and 0.7900, respectively (Table 19). Approximately 95% of the total score increase over the backbone is already achieved by SFT, while RL contributes an additional 0.0234. This does not imply that RL is ineffective: its additional improvement on MedVoice-Dial is 0.0968. Nevertheless, paired uncertainty estimates and targeted ablations are needed to establish the necessity of the proposed RL design.

Questions for authors
  1. What are the exact evaluation sizes and numbers of independent recordings, speakers/patients, and conversations in MedVoice-AID and MedVoice-Dial? How are policy SFT, both RL stages, judge training, and evaluation separated? Are all scenario variants, segments, and derivatives of the same source recording kept within the same partition?
  2. When the synthesized request is held fixed, how do acoustic-specific scores change if the clinical recording is removed, shuffled, or replaced with a matched normal recording? Do these controls demonstrate that correct acoustic evidence—not merely the request—drives the improvement?
  3. What are the contributions of the two RL stages individually and of the severity caps? Is the 0.0234 improvement over SFT on MedVoice-AID supported by paired confidence intervals, and does it exceed the benefit of additional SFT under a comparable budget?
  4. Please report negative-label prevalence and precision/recall for major and critical violations. Under independent physician annotation, does the full model reduce severe errors relative to SFT-only, rather than simply improve aggregate rubric coverage?
  5. Please provide paired confidence intervals for the main comparisons against the strongest open-weight baselines. In Table 6, were clean scores recomputed on the same 100-example subset used for noisy evaluation? The clean values exactly match Table 3, so clarification is needed to establish that clean/noisy comparisons use identical cases.
6 comments
AuthorsOpenReview ↗

Response to Reviewer D9wV(Part 1)

Q1. Evaluation sizes and data separation

We appreciate the reviewer’s attention to evaluation size and data separation. Both the number of independent observations and possible overlap matter for interpreting the results.

Q1.1. Evaluation sizes and independent units

Thank you for asking what counts as an independent observation. That distinction determines the appropriate resampling unit.

MedVoice-AID comprises 250 requests built from 98 distinct source recordings. MedVoice-Dial comprises 250 synthetic conversations. We use the source recording and the conversation, respectively, as the resampling units in Q5.1.

UnitMedVoice-AIDMedVoice-Dial
Evaluated examples250250
Source recordings98 source-audio families and 98 distinct source PCM hashesSynthetic speech
Audio files supplied250 composed request-plus-recording inputs623 distinct user-audio files including history
Conversations / requests250 distinct conversation IDs representing single-turn requests250 distinct synthetic conversations

The identity-mapping table covers 71 of the 98 AID source recordings. Identifiers are specific to each source.

Original sourceRecordings in AIDIdentity information recovered
ICBHI 20171311 patient IDs, matched to original demographic metadata
CirCor DigiScope1313 patient IDs, matched to the original patient table
Coswara1414 participant IDs, matched to original daily CSVs
SPRSound1311 recordings mapped to 11 patient IDs, while 2 remain unmapped
TORGO136 speaker IDs, traced to original speaker directories
VOICED55 record-associated IDs with original patient information files

Q1.2. Separation of policy training, judge training, and evaluation

We welcome the opportunity to clarify the training and evaluation boundaries, including which of them the available identifiers allow us to check.

The policy follows Qwen2.5-Omni-7B → medical SFT → dialogue RL RL1RL1 → audio-integrated RL RL2RL2 → benchmark inference. The two local reward judges are trained separately from GPT-5.4 item labels and supply rewards in their corresponding RL stage.

Stage / branchData and supervisionWhat is trained or evaluated
Medical SFT18,657 training examples: collected medical audio, synthesized spoken dialogues, and audio-integrated requests. Target responses provide supervisionAdapt the backbone to medical content and response requirements
Dialogue-judge trainingGPT-5.4 labels 112,514 training pairs, each matching a response to a criterion. Responses include outputs from GPT-4.1, GPT-4o-mini, and GPT-4.1-mini.Train a local Qwen3-4B judge using LoRA
RL16,927 dialogue examples from 1,941 conversations, each with dialogue history, current request, and associated rubricsStarting from SFT, sample response groups and update the policy with DAPO
Audio-judge trainingGPT-5.4 provides item labels for 22,872 training pairs drawn from current-policy responses and dialogue-training replay.Train a separate local Qwen3-4B audio-integrated judge using LoRA
RL24,546 audio-integrated requests from 600 recording families. Each request combines spoken input with clinical recordingsContinue from RL1 using DAPO and the audio-integrated judge
Benchmark evaluation250 examples each from AID and DialGenerate answers from saved policy checkpoints and score them with the external benchmark rubric scorer

We distinguish the documented construction rules from the identifier checks we could reproduce:

BoundaryConstruction rule / reproducible evidence
Dialogue judge train vs. held-outAppendix D.1 specifies disjoint conversations. Inspected files contain 1,950 training and 50 held-out conversation IDs
Audio judge train vs. held-outAppendix D.1 specifies source-family-disjoint splits and replay restricted to dialogue training data
RL1 train vs. validation1,941 vs. 56 conversations, with zero conversation-ID overlap
AID/Dial vs. SFTZero exact rendered-audio-path overlap with SFT train/validation
AID/Dial vs. RL1 and inspected judge splitsZero available sample/conversation-ID overlap with RL1 and zero conversation-ID overlap with judge splits
AID vs. RL2 trainingZero sample-ID, conversation-ID, recording-family, and source-PCM overlap
Dial vs. RL2 trainingZero sample-ID/conversation-ID overlap
AuthorsOpenReview ↗

Response to Reviewer D9wV(Part 2)

Q2. Fixed-request acoustic interventions

The proposed recording intervention is helpful, and we thank the reviewer for suggesting it. Keeping the request fixed makes the role of the acoustic evidence easier to examine.

We evaluated two conditions with the spoken request held fixed: silence and same-organ, opposite normal/abnormal-label recording replacement. The tables below follow the previously reported endpoint definitions and report all results as percentages.

DesignSilenceMismatched replacement
Clinical recordingSilenced while retaining the spoken requestReplaced by a same-organ recording from a different identifiable subject with the opposite label
Evaluation cohort100 sources103 paired sources from a separate expanded cohort
Policy inputNo reference answer or rubricNo reference answer or rubric
AssessmentContext-restored GPT-5.4 reviewContext-restored GPT-5.4 review

Table 1. Silence condition. The clinical recording is silenced while the spoken request is retained.

EndpointSFTFull
Acknowledges that audio evidence is insufficient for the task conclusion40.0%73.0%
Provides task-specific verification steps65.0%79.0%
Meets the complete cautious-response criterion12.0%10.0%

Under silence, Full more often acknowledged insufficient evidence 7373% vs\. 40% for SFT and suggested task-specific verification 7979% vs\. 65%. The complete cautious-response criterion was met in 10% of Full responses and 12% of SFT responses.

EndpointOperational definition
Insufficient task evidenceExplicitly states that the recording cannot establish the requested acoustic finding or task conclusion
Task-specific verificationAn actionable step: repeat recording/auscultation, relevant vital-sign checks, or a specified examination. Generic advice to seek care does not qualify
Complete cautious responseAdditionally withholds the audio-dependent conclusion and responds to the request, without unsupported acoustic assertions or a generic refusal

Table 2. Mismatched-recording replacement. Clinical audio remains available because a different recording accompanies the same spoken request.

EndpointSFTFull
Acknowledges that the recording alone is insufficient for the task conclusion55.3%83.5%
Provides task-specific verification steps62.1%78.6%
Makes the task decision conditional on further verification31.1%37.9%<br>
Combines evidence-boundary recognition, verification, and a response to the request35.0%68.0%
Strict label accuracy for the current replacement recording26.2%36.9%

With the replacement recording, Full met the composite criterion in 68.0% of cases, compared with 35.0% for SFT. Strict label accuracy on the replacement recording rose from 26.2% to 36.9%.

AuthorsOpenReview ↗

Response to Reviewer D9wV(Part 3)

Q3. Training stages, caps, and comparable-budget SFT

We appreciate the question about training stages, caps, and compute. Stagewise comparisons and a comparable-budget control address different aspects of where the gains arise.

The tables report scores at each training stage and differences between consecutive stages.

DatasetnBackbone Fig\. 5SFT Fig\. 5RL1Full Fig\. 5
MedVoice-AID2500.33410.76660.72900.7900
MedVoice-Dial2500.32370.42570.48230.5225
ACI-Bench5000.20020.21680.18400.2263
HealthDial2500.44890.41060.45970.5022
ThReadMed-QA2500.18000.20340.24670.3403
DatasetRL1 − SFTFull − RL1
AID-0.0376
−00˙727,−00˙031-0\.0727, -0\.0031

|+0.0610

+00˙348,+00˙871+0\.0348, +0\.0871

| |Dial|+0.0566

+00˙164,+00˙965+0\.0164, +0\.0965

|+0.0402

+00˙024,+00˙786+0\.0024, +0\.0786

| |ACI|-0.0329

−00˙639,−00˙019-0\.0639, -0\.0019

|+0.0423

+00˙186,+00˙662+0\.0186, +0\.0662

| |HealthDial|+0.0490

+00˙167,+00˙806+0\.0167, +0\.0806

|+0.0425

+00˙162,+00˙688+0\.0162, +0\.0688

| |ThReadMed|+0.0433

+00˙011,+00˙850+0\.0011, +0\.0850

|+0.0936

+00˙589,+01˙307+0\.0589, +0\.1307

|

  • RL1. RL1 improved on SFT on Dial, HealthDial, and ThReadMed-QA, while scores fell on AID and ACI-Bench.
  • RL2. After audio-integrated RL2, scores increased on all five tasks relative to RL1, with the largest gain on ThReadMed-QA +00˙936+0\.0936. Full had the highest point estimate of the four stages on each task.

Comparable-compute control. Both new arms start from the SFT checkpoint and use the same fixed training subsets. Each task has 100 fixed evaluation examples, scored with GPT-5.4.

Dataset (100 examples each)Continued SFTSmall-scale RL
MedVoice-AID0.68210.7672
MedVoice-Dial0.38840.4223
ACI-Bench0.21550.2253
HealthDial0.39170.4018
ThReadMed-QA0.15990.1560

At comparable modeled compute, small-scale RL scored 0.0851 higher than continued SFT on MedVoice-AID 0\.7672 vs\. 0\.6821. The absolute differences on the other four tasks ranged from 0.0039 to 0.0339.

Compute-control checkResult / interpretation
Modeled FLOPs gap: continued SFT relative to RLStage 1: −0.15% and stage 2: +0.71%

Caps-off ablation. RL2 starts from the RL1 checkpoint with the dangerous-error cap disabled and other reward terms retained.

TaskFull (paper)Caps-offFull − caps-off95% CI
MedVoice-AID0.79000.7095+0.0805
+00˙529,+01˙088+0\.0529, +0\.1088

| |MedVoice-Dial|0.5225|0.4573|+0.0652|

+00˙272,+01˙032+0\.0272, +0\.1032

| |ACI-Bench|0.2263|0.1800|+0.0463|

+00˙212,+00˙714+0\.0212, +0\.0714

| |HealthDial|0.5022|0.4206|+0.0816|

+00˙560,+01˙077+0\.0560, +0\.1077

| |ThReadMed-QA|0.3403|0.2159|+0.1244|

+00˙903,+01˙594+0\.0903, +0\.1594

| |MMedFD-Lite 1to101 to 10|5.780|5.408|+0.372|

+01˙16,+06˙20+0\.116, +0\.620

|

Full scored higher than caps-off on all six tasks, and all six 95% confidence intervals for the differences lay above zero.

Reviewer_HrJbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

MedSonor adapts Qwen2.5-Omni-7B to answer clinical requests that involve medical audio. Training combines three SFT sources (collected medical audio, GPT-5.4-synthesized spoken dialogues, and single-turn requests that splice real clinical recordings into TTS speech), followed by two DAPO stages. The RL reward comes from GPT-5.4-written instance rubrics, scored item by item by local Qwen3-4B judges, with severity caps for major and critical violations. The paper introduces two benchmarks (MedVoice-AID and MedVoice-Dial) and reports results on seven medical benchmarks, a 30-case physician preference study, an LLM-judged arena, general speech retention, and babble-noise robustness.

Critical strengths

Overall assessment The problem is important and the data effort is substantial. My score is driven by three issues, listed first under Concerns. The largest gain (MedVoice-AID) comes from a benchmark built and scored with the same pipeline as the training signal, the paper never tests whether the model uses the audio, and the physician study is too small to support the headline preference claim. If the authors show a large drop when audio is removed or swapped, describe MedVoice-AID's construction and its disjointness from training, and report the physician results with appropriate uncertainty, I would raise my score to 6. The remaining concerns would strengthen the paper but don't drive my rating. Strengths The paper targets a real gap. Existing medical audio work is mostly classification, and the authors correctly argue that recognizing a sound does not determine how to answer a clinical question about it (Section 2.2). The audio data assembly is broad (Tables 15 to 17) and is documented carefully. The authors honestly disclose that 349 Bridge2AI examples are Griffin–Lim reconstructions rather than original recordings. The local reward judges are validated against held-out GPT-5.4 labels, with accuracy reported separately for positive and negative items (Table 13). This level of reward-model transparency is uncommon and useful. The ablations (Figure 5, Table 19) separate the three SFT sources and compare SFT-only, RL-only and the full pipeline. The paper includes a human evaluation with practicing physicians, reports inter-rater agreement honestly, and checks retention on general speech tasks (Table 5).

Critical weaknesses

Concerns that drive my score

  1. The largest gain is on a benchmark that shares the training pipeline. GPT-5.4 writes the training rubrics and reference responses, labels the data for the local reward judges, and also scores MedVoice-AID with signed instance rubrics (Section 4.1). The paper does not describe how MedVoice-AID was built or whether it is disjoint from training at the level of source-audio family, scenario and template. Section 4.4 mentions 100 independent audio sources and Figure 4 mentions 47 case families, which suggests related items exist. The results pattern points to a pipeline effect rather than general capability. On MedVoice-AID, MedSonor scores 0.7900 against 0.5579 for Gemini-3.7-Flash and 0.5319 for GPT-Audio (Table 3). On every other benchmark, including the authors' own MedVoice-Dial (0.5225 against 0.7126), MedSonor trails both of those closed models (Tables 3 and 4). Even the variant trained without audio-integrated SFT data scores 0.5957 on MedVoice-AID (Table 19), still above every closed model. Gemini-2.5-Flash scores only 0.1676 there, below several 7B open models. Both observations suggest the benchmark rewards pipeline-specific response style.
  2. Nothing tests whether the model actually uses the audio. Both the local reward judge and the benchmark judge receive the audio content as text, so neither checks acoustic grounding directly. A model could earn high rubric scores by producing well-hedged responses in the expected style without attending to the recording. An evaluation with silenced, shuffled or mismatched clinical audio on MedVoice-AID would settle this. If scores drop sharply, the central claim is supported. If they barely move, the gains are about response style.
  3. The physician study does not support "ranks highest in aggregate physician preference." The study is valuable, but its size limits what it can show. It covers 30 cases, and Figure 3's intervals for MedSonor (1132.3), MiniCPM-o (1100.3) and Baichuan (1087.4) appear to overlap. Direct preference credit against MiniCPM-o is 53.4% and against Qwen2.5-Omni is 55.0%, close to chance. Pairwise agreement among physicians is 59.6% (95% interval 46.0 to 73.8%, Appendix F). Physicians prefer MedSonor far less decisively than the Gemini arena judge does (for example 53.4% against MiniCPM-o versus 69.5% in Figure 4), and far less than the 2x rubric-score gap implies. The paper should state this gap and discuss what it means, rather than presenting the human study as confirming the rubric results. Additional concerns
  4. The audio-integrated ablation is not a clean ablation. The "w/o audio-integrated dialogue" variant removes only the SFT data, while the audio-integrated RL stage still runs so the result doesn't isolate the contribution of pairing audio with requests.
  5. Baselines are missing. SpeechMedAssist and StethoLM are discussed (Table 2, Section 5) but not compared. There is no cascade baseline, such as ASR plus an audio description model feeding a strong medical text LLM, which would show what end-to-end training adds. The open baselines do not include a current omni model such as Qwen3-Omni.
  6. Several margins are small and no variance is reported. On LiveMedBench, MedSonor scores 0.2843 against 0.2803 for Baichuan, and on MMedFD-Lite it scores 5.780 against 5.692 (Table 3). No confidence intervals or seed variance accompany the benchmark tables, so "ranks first" on these two benchmarks isn't established.
  7. Response length may confound rubric scores. The RL length penalty is set relative to the GPT-5.4 reference length (Appendix C), and coverage-based rubrics tend to reward longer answers. Reporting response-length distributions for all models, and a length-controlled comparison, would rule this out.
  8. The RL stage looks fragile on its own. RL-only falls below the base model on 4 of 5 tasks, and SFT alone lowers HealthDial from 0.4489 to 0.4106 (Table 19). The paper would benefit from a short analysis of why RL without SFT degrades performance.
  9. The severity caps are not ablated, and safety is not evaluated directly. The caps are a stated contribution, but there is no run without them. The audio judge's negative-item accuracy is 85.77%, against 99.39% for the dialogue judge (Table 13), so the signal for prohibited errors is weakest exactly where audio is involved.

Minor issues The noise robustness table (Table 6) omits MiniCPM-o, which is the strongest baseline in the human study. LLaMA-Omni scores higher under moderate noise than clean, which deserves a comment. General speech retention is mostly good, but LibriSpeech WER rises from 2.26 to 2.78 (about 23% relative), which should be stated plainly (Table 5). Technical novelty is mainly the combination of existing components (Rubrics-as-Rewards, DAPO and SFT) with severity caps.

Questions for authors

How were MedVoice-AID and MedVoice-Dial constructed, and are they disjoint from training by source-audio family, scenario and template? What are MedSonor's and the baselines' MedVoice-AID scores when the clinical audio is silenced, shuffled across items, or replaced with mismatched recordings? Can you report confidence intervals for the physician preferences against each baseline and explain the gap between physician and arena preferences? What happens on the external benchmarks when a judge from a different model family scores them? What are the mean response lengths per model, and do the rankings hold under a length-controlled comparison? What does an ablation without the severity caps show?

15 comments
Reviewer_HrJbOpenReview ↗

I was urged by Program chairs to give another review so I am revising my review of the paper here

Summary:

The paper addressed the problem of how medical voice assistants can use clinical audio (lung, heart, cough, voice-disorder sounds) when answering requests. The data is sourced from three sources: collected medical audio, about 7.5k examples from many public datasets, GPT-5.4 multi-turn dialogues turned into TTS, built from 25 specialties x 15 scenarios and audio-integrated requests, with a TTS request and a real clip spliced into it. To train their network, the authors use SFT, followed by dialogue RL and audio-integrated RL. A local Qwen3-4B judge trained on GPT-5.4 labels is used to score responses against the rubric, such that responses with major or critical errors cannot receive a high reward. Further, they introduce two new benchmarks, MedVoice-AID and MedVoice-Dial. The model is evaluated on 7 medical benchmarks, a 6-physician preference study of 30 cases. A general speech retention check and babble-noise robustness study is also presented.

Strengths:

  1. Data coverage is broad, spanning many audio types across respiratory, cardiac etc.
  2. The rubric design is careful, with history constraints and severity caps meant to stop fluent answers from hiding serious errors.
  3. The evaluation is wide, with many benchmarks plus physician and LLM preference studies.
  4. The ablations are informative, especially SFT vs RL-only and removing the audio-integrated data.
  5. Limitations are reported honestly, including the Griffin–Lim reconstruction artefacts, GPT-5.4 dependence and English-only data.
  6. The method is reproducible in detail, with prompt templates, hyperparameters, data counts and source tables in the appendices.

Weaknesses:

  1. None of the ablations isolate the RL stages. There’s no dialogue-RL-only, no audio-RL-only and no single joint RL stage. The paper never shows that the sequential design matters, or what each stage adds on its own.

  2. It is not clear how sequential rubric guided RL is part of the pipeline from the text or figures. As per my understanding, the pipeline has three stages in a row. SFT comes first. Then there is RL on the synthesized multi-turn dialogues (6,927 examples) using the dialogue judge. Then a second RL stage on the audio-integrated requests (4,546 examples) using a separate audio judge. Each RL stage starts from the previous stage’s checkpoint and applies KL against that checkpoint. However, Figure 2 draws only one generic Stage 2 RL block, so the two-stage ordering is never shown.

  3. In the audio integrated case, no real user records a question. The obvious alternative is a text or separate spoken request plus a separate audio input, and the authors never compare against it. MedVoice-AID is built with the same composition, so training and test share this artificial format.

  4. Every spoken part used in RL is TTS and doesn’t cover real recording conditions or speaker variation. In audio-integrated RL, only the spliced-in clinical clip is real. Since evaluations are also TTS, train and test share the TTS voice distribution. MMedFD-Lite is the only real-speech benchmark, and the gain there is small.

  5. Since the judges never hear the audio, the reward therefore checks consistency with a GPT-written description of the clip, not with the sound itself.

  6. Table 5 doesn’t support the conclusion that general speech ability is broadly retained. Since medical fine tuning is producing considerable gains on general tasks, it points to an evaluation mismatch for the backbone, such as a judge bias, rather than real retention. Further, the tests is a single run with no variance.

Questions:

  1. In the integrated audio case, why was the request and the clinical clip are merged into one audio stream instead of being given as separate inputs. How will a multi-channel input change things?
  2. Why does AISHELL, a mandarin dataset improves after English-only training?
  3. How was the MedVoice-AID and MedVoice-Dial test sets are kept separate from the training data?
  4. What were the STFT parameters, number of Griffin–Lim iterations, and whether the spectrograms were mel or linear?
  5. Does a speech-enhancement front end improve robustness in the presence of babble noise, or does it remove clinically relevant non-speech sounds from the composed input?

My rating of the paper remains unchanged as of 11/14/26

AuthorsOpenReview ↗

Response to Reviewer HrJb’s Initial Review(Part 1)

W1. Benchmark construction, overlap, and pipeline-specific gains

We appreciate the question about benchmark construction and training gains. To interpret those gains, readers need to know which examples share a source.

W1.1. Evaluation sizes and independent units

Thank you for drawing attention to the distinction between requests and independent observations. It matters for interpreting the confidence intervals.

  • MedVoice-AID contains 250 requests derived from 98 source recordings. MedVoice-Dial contains 250 synthetic conversations.
  • To estimate 95% confidence intervals for differences in model performance, we use 20,000 paired bootstrap resamples.
  • For MedVoice-AID, we sample source recordings with replacement and include all request variants from each selected recording. For MedVoice-Dial, we sample whole conversations with replacement.
UnitMedVoice-AIDMedVoice-Dial
Evaluated examples250250
Source recordings98 source-audio families and 98 distinct source PCM hashesSynthetic speech
Audio files supplied250 composed request-plus-recording inputs623 distinct user-audio files including history
Conversations / requests250 distinct conversation IDs representing single-turn requests250 distinct synthetic conversations

The sources listed below account for 71 of the 98 AID recordings. Patient and speaker identifiers are source-specific.

Original sourceRecordings in AIDIdentity information recovered
ICBHI 20171311 patient IDs, matched to original demographic metadata
CirCor DigiScope1313 patient IDs, matched to the original patient table
Coswara1414 participant IDs, matched to original daily CSVs
SPRSound1311 recordings mapped to 11 patient IDs, while 2 remain unmapped
TORGO136 speaker IDs, traced to original speaker directories
VOICED55 record-associated IDs with original patient information files
AuthorsOpenReview ↗

Response to Reviewer HrJb’s Initial Review(Part 2)

W1.2. Separation of policy training, judge training, and evaluation

We welcome the opportunity to clarify how the datasets were separated, particularly which boundaries we could verify from the available identifiers.

The policy follows Qwen2.5-Omni-7B → medical SFT → dialogue RL (RL1) → audio-integrated RL (RL2) → evaluation. Two separately trained local judges use GPT-5.4 criterion-level labels and provide rewards for the corresponding RL stages.

Stage / branchData and supervisionWhat is trained or evaluated
Medical SFT18,657 training examples: collected medical audio, synthesized spoken dialogues, and audio-integrated requests. Target responses provide supervisionAdapt the backbone to medical content and response requirements
Dialogue-judge trainingGPT-5.4 labels individual pairings of responses with criteria. Responses include GPT-4.1, GPT-4o-mini, and GPT-4.1-mini. The training set contains 112,514 pairsTrain a local Qwen3-4B judge using LoRA
RL16,927 dialogue examples from 1,941 conversations, each with dialogue history, current request, and associated rubricsStarting from SFT, sample response groups and update the policy with DAPO
Audio-judge trainingCurrent-policy responses plus dialogue-training replay with GPT-5.4 item labels across 22,872 training pairsTrain a separate local Qwen3-4B audio-integrated judge using LoRA
RL24,546 audio-integrated requests from 600 recording families, each combining a spoken request with a clinical recordingContinue from RL1 using DAPO and the audio-integrated judge
Benchmark evaluation250 examples for AID and 250 for DialGenerate answers from saved policy checkpoints and score them with the external benchmark rubric scorer

The documented construction rules and available overlap checks are:

BoundaryConstruction rule / reproducible evidence
Dialogue judge train vs. held-outAppendix D.1 specifies disjoint conversations. Inspected files contain 1,950 training and 50 held-out conversation IDs
Audio judge train vs. held-outAppendix D.1 specifies source-family-disjoint splits and replay restricted to dialogue training data
RL1 train vs. validation1,941 vs. 56 conversations, with zero conversation-ID overlap
AID/Dial vs. SFTZero exact rendered-audio-path overlap with SFT train/validation
AID/Dial vs. RL1 and inspected judge splitsZero available sample/conversation-ID overlap with RL1 and zero conversation-ID overlap with judge splits
AID vs. RL2 trainingZero sample-ID, conversation-ID, recording-family, and source-PCM overlap
AID vs. RL2 fixed validation110 evaluation examples overlap, spanning 39 recording families. The training-set separation above does not extend to this validation set
Dial vs. RL2 trainingZero sample-ID/conversation-ID overlap

AID and RL2 training share no recording families or source PCM hashes. The fixed RL2 validation set overlaps with 110 AID examples from 39 recording families. For SFT, the overlap check compared exact rendered-audio paths.

W1.3. Scenario and template construction

The reviewer raises a useful question about template reuse. We appreciate it because the scope of generalization depends on which structures training and evaluation share.

  • AID combines a user role, care setting, and requested action with a source recording and acoustic annotations to generate a spoken request, reference answer, and instance-specific signed rubric. One recording may support multiple requests.

  • Its 250 examples span 15 scenario categories, all shared with audio-integrated RL. SFT and RL also use this scenario inventory.

  • Dial generates a topic from a department and scenario, expands it into four exchanges between the user and the assistant, and evaluates the current turn using its history, reference, rubric, and synthesized user speech. The 250 conversations span 25 departments and 15 scenario categories.

  • During construction, we excluded existing topics from the inspected SFT and dialogue-RL sources and screened for exact and lexical similarity.

LevelConstruction and separation
Scenario categoriesBroad roles, settings, and task types are reused, so the benchmarks are not evaluations on held-out scenario categories.
Generation templatesShared prompt structures specify the clinical context, response format, and rubric schema. These structures are not held out between training and evaluation, although individual generation instructions differ across builds.

We will describe AID and Dial as evaluations within a shared task-generation framework.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.