- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Evidence-to-Capability Matching (ECM) for medical training data synthesis. The method extracts target capabilities from reference questions, matches them with medical sources, and uses MedPAR to refine and verify the generated questions under source-specific constraints.
The authors construct three task-focused datasets and train Qwen3 models with GRPO. On Qwen3-8B, they report improvements of 2.69, 6.68, and 1.46 percentage points on MedXpertQA, MedQA, and MedMCQA, respectively. The paper also includes guidance ablations, cross-task evaluations, and experiments on Qwen3-30B-A3B.
Critical strengths
-
The distinction between medical correctness and capability alignment is useful. A question can be factually correct without testing the intended reasoning skill. The source-specific constraints also help prevent generated questions from making claims that the original medical evidence cannot support.
-
The experimental results are promising, especially the 6.68-point improvement on MedQA. The six guidance ablations provide additional evidence that the construction requirements matter, and the paired statistical tests strengthen the clinical-decision results. The improvements on Qwen3-30B-A3B are also encouraging, although they cover fewer settings.
Critical weaknesses
-
The experiments do not sufficiently isolate the contribution of ECM. There is no direct comparison with a simpler medical question generation pipeline under the same GRPO settings, data size, and compute budget. The guidance ablations are useful, but they only remove individual components within ECM, and the resulting datasets and selected checkpoints can differ. One result that stands out is Table 3: removing Assessment Interface reduces MedQA accuracy by 5.34 points, close to the 5.81-point drop from removing Knowledge Target. This suggests that question format and task-distribution matching may explain a considerable part of the improvement. Without a simpler, format-matched baseline, it is difficult to determine how much benefit comes from the proposed matching process itself.
-
The evaluation protocol raises concerns about independent generalization. Appendix C.2 states that the reported primary benchmarks are used for checkpoint selection. HLE_verified is used for ER, while the four-task average is used for CT. Although HLE questions are excluded from synthesis and training, they still influence model selection. The reported 2.79-point gain on HLE also corresponds to only six additional correct answers out of 215 questions. The data used for capability profiling needs clarification as well. PROFILE can use question stems, answer options, and reference answers, but the paper does not clearly establish which splits these reference questions come from. If evaluation questions were involved, the capability profiles could contain information about the test distribution. I am not claiming this occurred, but the current description does not rule it out.
Questions for authors
-
Can you compare ECM against a simpler source-grounded question generation pipeline using the same medical sources, generation model, data size, question format, and GRPO training budget?
-
Can you further isolate the contributions of capability profiling, source alignment, and MedPAR? The current guidance ablations do not directly evaluate these main stages of the pipeline.
-
Which dataset splits were used to construct the capability profiles? Were any evaluation questions, answer options, or reference answers used during profiling or source matching?
-
Do the reported improvements hold when checkpoints are selected using separate validation data or fixed training steps? How stable are the results across training seeds, particularly on HLE_verified?
-
Was the final synthetic training data independently evaluated by medical experts? If so, what fraction of the accepted questions were judged medically correct, supported by their sources, and aligned with the intended capabilities?