acceptodds
Under review as a conference paper at ICLR 2027

Evidence-to-Capability Matching for Medical Training Data Synthesis

Abstract

Synthetic training data for medical large language models (LLMs) draws on heterogeneous sources, but topical relevance does not guarantee that a source supports the capability a target task requires. We introduce Evidence-to-Capability Matching (\method), a framework that profiles target capabilities, identifies source-supported evidence, and specifies requirements for constructing training questions. Its refinement component, Medical Profile-Routed Agentic Refinement (\refiner), coordinates source-constrained revision and verification for medical validity and capability alignment. On Qwen3-8B, task-focused corpora improve their primary benchmarks for evidence-intensive reasoning, clinical decisions, and knowledge coverage by 2.69, 6.68, and 1.46 percentage points, respectively. Ablations of six guidance families show that full guidance performs best in both tested settings, while the most consequential guidance differs by task: knowledge targeting for clinical decisions and evidence structuring for evidence-intensive reasoning. Clinical-decision-oriented training also improves on a benchmark excluded from synthesis and training, and a cross-task mixture improves over the base model on all four benchmarks. These results suggest that effective medical supervision depends on matching what a task requires to what its sources can support.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Data-Centric AI, Agentic Refinement

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_2Dd4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Evidence-to-Capability Matching (ECM) for medical training data synthesis. The method extracts target capabilities from reference questions, matches them with medical sources, and uses MedPAR to refine and verify the generated questions under source-specific constraints.

The authors construct three task-focused datasets and train Qwen3 models with GRPO. On Qwen3-8B, they report improvements of 2.69, 6.68, and 1.46 percentage points on MedXpertQA, MedQA, and MedMCQA, respectively. The paper also includes guidance ablations, cross-task evaluations, and experiments on Qwen3-30B-A3B.

Critical strengths
  1. The distinction between medical correctness and capability alignment is useful. A question can be factually correct without testing the intended reasoning skill. The source-specific constraints also help prevent generated questions from making claims that the original medical evidence cannot support.

  2. The experimental results are promising, especially the 6.68-point improvement on MedQA. The six guidance ablations provide additional evidence that the construction requirements matter, and the paired statistical tests strengthen the clinical-decision results. The improvements on Qwen3-30B-A3B are also encouraging, although they cover fewer settings.

Critical weaknesses
  1. The experiments do not sufficiently isolate the contribution of ECM. There is no direct comparison with a simpler medical question generation pipeline under the same GRPO settings, data size, and compute budget. The guidance ablations are useful, but they only remove individual components within ECM, and the resulting datasets and selected checkpoints can differ. One result that stands out is Table 3: removing Assessment Interface reduces MedQA accuracy by 5.34 points, close to the 5.81-point drop from removing Knowledge Target. This suggests that question format and task-distribution matching may explain a considerable part of the improvement. Without a simpler, format-matched baseline, it is difficult to determine how much benefit comes from the proposed matching process itself.

  2. The evaluation protocol raises concerns about independent generalization. Appendix C.2 states that the reported primary benchmarks are used for checkpoint selection. HLE_verified is used for ER, while the four-task average is used for CT. Although HLE questions are excluded from synthesis and training, they still influence model selection. The reported 2.79-point gain on HLE also corresponds to only six additional correct answers out of 215 questions. The data used for capability profiling needs clarification as well. PROFILE can use question stems, answer options, and reference answers, but the paper does not clearly establish which splits these reference questions come from. If evaluation questions were involved, the capability profiles could contain information about the test distribution. I am not claiming this occurred, but the current description does not rule it out.

Questions for authors
  1. Can you compare ECM against a simpler source-grounded question generation pipeline using the same medical sources, generation model, data size, question format, and GRPO training budget?

  2. Can you further isolate the contributions of capability profiling, source alignment, and MedPAR? The current guidance ablations do not directly evaluate these main stages of the pipeline.

  3. Which dataset splits were used to construct the capability profiles? Were any evaluation questions, answer options, or reference answers used during profiling or source matching?

  4. Do the reported improvements hold when checkpoints are selected using separate validation data or fixed training steps? How stable are the results across training seeds, particularly on HLE_verified?

  5. Was the final synthetic training data independently evaluated by medical experts? If so, what fraction of the accepted questions were judged medically correct, supported by their sources, and aligned with the intended capabilities?

Reviewer_Fb8KOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes ECM, a pipeline designed to generate medical synthetic data that targets specific skills from questions using six guidelines linking to clinical reasoning skills using LLM agents.

Critical strengths
  1. The motivation is reasonable.
Critical weaknesses
  1. The main claim and design are not well supported. The paper asserts that existing synthetic data can be correct yet fail to train the target skill, but it never demonstrates this limitation in any prior method or dataset. The six guidelines are also introduced without citation or justification, and they intentionally overlap by design (App. F). Additionally, the evaluation lacks proper baselines: the fine-tuned model is only compared against the base model rather than models trained on alternative synthetic datasets. It is therefore unclear how much of the performance gain stems from ECM itself versus standard GRPO on in-domain questions.

  2. The evaluation is unreliable. Checkpoints are selected directly using performance on test benchmarks, which leads to data leakage. Furthermore, the underlying data sources are not reported. The ablation study is also weak: while it shows performance drops when removing a guidance family, it fails to control for dataset size, making it impossible to tell whether the drop is driven by having less overall training data or losing that specific category.

Questions for authors
  1. Could the authors add more examples and clarification to the introduction? The current paragraph is difficult to read, making the key motivating points hard to understand.

  2. How were the six guidances chosen? Are they based on prior work (e.g., medical item-writing guidelines), derived empirically from data, or validated in any way?

  3. A clinician evaluation of a sample of generated questions would help demonstrate that they are medically correct and genuinely test the intended skill.

Reviewer_EGPDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes Evidence-to-Capability Matching (ECM), a framework for synthesizing high-quality medical training data to enhance language model capabilities. Its main contribution is an integrated pipeline combining capability-guided data generation, source-grounded question construction, and agentic refinement and verification. The approach is validated by comparing the original Qwen3 model with its ECM-trained counterpart, demonstrating the usefulness of the synthesized data.

Critical strengths
  1. The paper addresses an important problem in medical data synthesis. It highlights that medically correct training examples do not necessarily target the intended capability, which is an important distinction for training medical LLMs.

  2. The proposed framework provides a clear and structured workflow. Separating capability requirements from source-supported evidence, and enforcing both during refinement, offers a practical approach to reducing unsupported claims and task misalignment.

  3. The experiments cover multiple medical capabilities and evaluation benchmarks. The paper evaluates evidence-intensive reasoning, clinical decision-making, and knowledge coverage, and also investigates cross-task transfer and performance on a larger model.

Critical weaknesses
  1. The novelty is somewhat limited. The main contribution of this work is a data synthesis pipeline that combines several established techniques, including capability-guided data generation, source-grounded question construction and agentic refinement and verification. While their integration is reasonable, similar components have been explored in prior work.
  2. The main experiments lacks strong baselines. Table 1 primarily compare the original Qwen3 model against the same model trained on ECM-generated data. Although these results demonstrate the utility of the synthesized training data, is remains uncertain whether ECM outperforms simpler data generation approaches or existing synthesis pipelines. Furthermore, the paper lacks a Related Work section, making it difficult to assess how the proposed approach differs from and improves upon the existing methods.
  3. The data synthesis procedure lacks reproducibility. The paper does not clearly identify the models used for the writer and verifier components in the data synthesis pipeline. Additionally, the author does not specify whether these models are trained or not.
  4. The paper contains several minor formatting and citation errors. For example, line 719 contains the unresolved reference “Sections ??–??”. Additionally, the citations in lines 229–230 are improperly formatted, as in “Qwen3-8B Yang et al. (2025) and uses GRPO Shao et al. (2024),” .
Questions for authors

none

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.