- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MedSonor adapts Qwen2.5-Omni-7B for medical audio dialogue through supervised fine-tuning and two stages of rubric-guided RL. The training data combine medical recordings, synthesized spoken consultations, and clinical requests paired with recordings. The paper also introduces MedVoice-Dial and MedVoice-AID. MedSonor outperforms the evaluated general-purpose open-weight baselines on seven medical response benchmarks and receives the highest aggregate physician preference score on 30 cases. Ablations, general speech evaluations, and a noise experiment provide additional analysis of the training pipeline.
Critical strengths
-
The paper addresses a useful problem: recognizing a clinical sound is only part of answering a medical question about it. The combination of recordings, spoken dialogue, and audio-integrated requests in Section 3.2 is well motivated and covers different aspects of this task.
-
The source ablations help explain where the gains come from. In particular, removing audio-integrated SFT lowers MedVoice-AID from 0.7900 to 0.5957 even though both RL stages are retained (Table 19). This is a substantial difference and supports the value of the proposed data construction.
-
I appreciate the effort to evaluate beyond the training rubric. The blinded physician study and the separate audio-capable pairwise evaluator provide useful checks on the automated scores (Figures 3–4 and Appendix F). Results on adapted public datasets also suggest that the improvements are not confined to the two new benchmarks.
Critical weaknesses
-
The train/test separation of MedVoice-AID and MedVoice-Dial is not clear enough. Appendix D.1 describes disjoint splits for the local judges, but this does not establish that the policy's evaluation recordings and conversations are disjoint from all SFT and RL sources. Since the same recording can produce several scenario variants, splitting generated examples is not sufficient. Please give the test sizes, grouping rules, and overlap counts at the recording, patient/speaker where available, and conversation level. This is particularly important for MedVoice-AID, where the largest gain is reported.
-
I am not yet convinced that the improvement measures better use of clinical acoustic evidence. GPT-5.4 generates the cases, references, and rubric supervision and also scores the main medical evaluations; the local audio judge receives reference text rather than the waveform (Appendices D.1 and G). A useful control would keep the spoken request fixed and remove or replace the clinical recording, then check whether the answer changes appropriately. Without this, it is hard to separate better acoustic interpretation from better use of contextual cues and the expected answer format.
-
The contribution of the severity caps is not established by the current ablations. SFT alone already scores 0.7666 on MedVoice-AID, compared with 0.7900 for the full pipeline (Table 19), and there is no matched comparison with the caps disabled. Since avoiding serious medical errors motivates this part of the method, I would like to see its effect on independently assessed major/critical-error rates, together with uncertainty on the SFT-to-RL improvement.
-
Some evaluation claims need more qualification. The physician study covers only 30 cases, and the direct preference credit against MiniCPM-o is 53.4%, so the highest aggregate rank should not be read as a clear advantage over every comparator. Also, Section 4.4 uses a 100-source subset for the noise experiment, while Table 6's clean scores exactly repeat Table 3. Please clarify whether the clean and noisy scores use the same examples; otherwise, their difference is not a matched estimate of noise sensitivity.
Questions for authors
See weaknesses 1–4.
7 comments
Response to Reviewer GjwS (Part 2)
W1.2. Separation of policy training, judge training, and evaluation
We welcome the request to clarify data separation across stages, particularly the distinction between documented construction rules and checks we could reproduce.
The policy follows Qwen2.5-Omni-7B → medical SFT → dialogue RL (RL1) → audio-integrated RL (RL2) → benchmark inference. The two local reward judges are trained separately using GPT-5.4 criterion-level labels and provide rewards during their respective RL stages.
| Stage / branch | Data and supervision | What is trained or evaluated |
|---|---|---|
| Medical SFT | 18,657 training examples: collected medical audio, synthesized spoken dialogues, and audio-integrated requests. Target responses provide supervision | Adapt the backbone to medical content and response requirements |
| Dialogue-judge training | GPT-5.4 labels individual responses paired with criteria. Responses include GPT-4.1, GPT-4o-mini, and GPT-4.1-mini. The training set contains 112,514 pairs | Train a local Qwen3-4B judge using LoRA |
| RL1 | 6,927 dialogue examples from 1,941 conversations, each with dialogue history, current request, and associated rubrics | Starting from SFT, sample response groups and update the policy with DAPO |
| Audio-judge training | Current-policy responses plus dialogue-training replay with GPT-5.4 item labels across 22,872 training pairs | Train a separate local Qwen3-4B audio-integrated judge using LoRA |
| RL2 | 4,546 audio-integrated requests from 600 recording families, each combining a spoken request with a clinical recording | Continue from RL1 using DAPO and the audio-integrated judge |
| Benchmark evaluation | AID: 250 examples, Dial: 250 examples | Generate answers from saved policy checkpoints and score them with the external benchmark rubric scorer |
The following table distinguishes the documented data-construction rules from the overlap checks supported by the available identifiers.
| Boundary | Construction rule / reproducible evidence |
|---|---|
| Dialogue judge train vs. held-out | Appendix D.1 specifies disjoint conversations. Inspected files contain 1,950 training and 50 held-out conversation IDs |
| Audio judge train vs. held-out | Appendix D.1 specifies source-family-disjoint splits and replay restricted to dialogue training data |
| RL1 train vs. validation | 1,941 vs. 56 conversations, with zero conversation-ID overlap |
| AID/Dial vs. SFT | Zero exact rendered-audio-path overlap with SFT train/validation |
| AID/Dial vs. RL1 and inspected judge splits | Zero available sample/conversation-ID overlap with RL1 and zero conversation-ID overlap with judge splits |
| AID vs. RL2 training | Zero sample-ID, conversation-ID, recording-family, and source-PCM overlap |
| Dial vs. RL2 training | Zero sample-ID/conversation-ID overlap |
Response to Reviewer GjwS (Part 3)
W2. Fixed-request controls for clinical acoustic evidence
We appreciate the reviewer’s question about changes to the clinical recording. Keeping the spoken request constant allows us to examine whether the answer responds to those changes.
We held the spoken request fixed and either silenced the clinical recording or replaced it with a same-organ recording carrying the opposite normal/abnormal label. All outcome values are percentages.
| Design | Silence | Mismatched replacement |
|---|---|---|
| Clinical recording | Silenced while retaining the spoken request | Replaced by a same-organ recording from a different identifiable subject with the opposite label |
| Evaluation cohort | 100 sources | 103 paired sources from a separate expanded cohort |
| Policy input | No reference answer or rubric | No reference answer or rubric |
| Assessment | Context-restored GPT-5.4 review | Context-restored GPT-5.4 review |
Table 1. Silence condition. The clinical recording is silenced while the spoken request is retained.
| Endpoint | SFT | Full |
|---|---|---|
| Acknowledges that audio evidence is insufficient for the task conclusion | 40.0% | 73.0% |
| Provides task-specific verification steps | 65.0% | 79.0% |
| Meets the complete cautious-response criterion | 12.0% | 10.0% |
Under silence, Full more often acknowledged insufficient evidence (73% vs. 40% for SFT) and suggested task-specific verification (79% vs. 65%). The complete cautious-response criterion was met in 10% of Full responses and 12% of SFT responses.
| Endpoint | Operational definition |
|---|---|
| Insufficient task evidence | Explicitly states that the recording cannot establish the requested acoustic finding or task conclusion |
| Task-specific verification | An actionable step: repeat recording/auscultation, relevant vital-sign checks, or a specified examination. Generic advice to seek care does not qualify |
| Complete cautious response | Additionally withholds the audio-dependent conclusion and responds to the request, without unsupported acoustic assertions or a generic refusal |
Table 2. Mismatched-recording replacement. Clinical audio remains available because only the recording is replaced, with the request unchanged.
| Endpoint | SFT | Full |
|---|---|---|
| Acknowledges that the recording alone is insufficient for the task conclusion | 55.3% | 83.5% |
| Provides task-specific verification steps | 62.1% | 78.6% |
| Makes the task decision conditional on further verification | 31.1% | 37.9% |
| Combines evidence-boundary recognition, verification, and a response to the request | 35.0% | 68.0% |
| Strict label accuracy for the current replacement recording | 26.2% | 36.9% |
With the spoken request held fixed, strict label accuracy for the replacement recording was 36.9% for Full and 26.2% for SFT. Full also more often acknowledged the limits of the evidence, proposed verification, and addressed the request (68.0% vs. 35.0%).
W3. Severity caps, SFT-to-RL uncertainty, and severe errors
Thank you for asking about both stagewise gains and severe errors. We agree that the contribution of training and the safety of the resulting responses deserve separate examination.
W3.1. Training stages and stagewise confidence intervals
We appreciate the request for a stage-by-stage account. Paired differences between successive stages show the size and uncertainty of each change.
The following tables report scores at each training stage and differences between consecutive stages. Intervals in brackets are 95% confidence intervals. In W3.1 and W3.2, differences were computed before rounding.
| Dataset | n | Backbone (Fig. 5) | SFT (Fig. 5) | RL1 | Full (Fig. 5) |
|---|---|---|---|---|---|
| MedVoice-AID | 250 | 0.3341 | 0.7666 | 0.7290 | 0.7900 |
| MedVoice-Dial | 250 | 0.3237 | 0.4257 | 0.4823 | 0.5225 |
| ACI-Bench | 500 | 0.2002 | 0.2168 | 0.1840 | 0.2263 |
| HealthDial | 250 | 0.4489 | 0.4106 | 0.4597 | 0.5022 |
| ThReadMed-QA | 250 | 0.1800 | 0.2034 | 0.2467 | 0.3403 |
| Dataset | RL1 − SFT | Full − RL1 |
|---|---|---|
| AID | -0.0376 [-0.0727, -0.0031] | +0.0610 [+0.0348, +0.0871] |
| Dial | +0.0566 [+0.0164, +0.0965] | +0.0402 [+0.0024, +0.0786] |
| ACI | -0.0329 [-0.0639, -0.0019] | +0.0423 [+0.0186, +0.0662] |
| HealthDial | +0.0490 [+0.0167, +0.0806] | +0.0425 [+0.0162, +0.0688] |
| ThReadMed | +0.0433 [+0.0011, +0.0850] | +0.0936 [+0.0589, +0.1307] |
- Dialogue RL. RL1 scored higher than SFT on MedVoice-Dial, HealthDial, and ThReadMed-QA, and lower on MedVoice-AID and ACI-Bench.
- Full policy. Full scored higher than RL1 on all five tasks, with the largest difference on ThReadMed-QA (+0.0936), and had the highest point estimate among the four stages on each task.
Response to Reviewer GjwS (Part 1)
We thank the reviewer for the careful assessment and constructive suggestions. We address each concern below.
W1. Evaluation sizes, grouping, and train/test separation
We appreciate the question about grouping and training overlap. We agree that the grouping unit and the limits of what the identifiers can establish should both be clear.
We checked the policy evaluation sets against the available training identifiers. For AID versus RL2 training, we found no overlap in recording families or source PCM. For SFT, we checked for exact matches in rendered audio paths.
W1.1. Evaluation sizes and independent units
Thank you for distinguishing the number of evaluation examples from the number of independent units. The latter matters for estimating uncertainty.
The following table lists the available identity information for sources accounting for 71 of the 98 AID recordings. Identifiers are defined within each source and should not be interpreted as globally unique patient or speaker identifiers.