- Rating
- 3(+1)
- Confidence
- 3(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes CoEvoMed, a framework that couples medical vision-language model training with multi-agent supervision construction. It revises supervision principles and adjusts curriculum allocation using feedback on the learner’s evolving capabilities. The authors evaluate the framework on 2D medical VQA, report generation, and 3D medical tasks, reporting improvements over several baselines and presenting interventions examining feedback alignment and stage-dependent principle utility.
Critical strengths
-
Stage-dependent supervision is supported by controlled comparisons. Applying original and evolved construction principles at different checkpoints reveals a reversal in their relative effectiveness. This provides concrete evidence that supervision should adapt to the learner’s training stage, supporting a central motivation of the framework.
-
Broad medical multimodal evaluation. The experiments cover 2D visual question answering, report generation, and 3D tasks, with additional backbone evaluations. This breadth provides useful evidence that the framework can apply across multiple medical task formats and model architectures.
Critical weaknesses
-
The experimental controls and mechanism analyses do not sufficiently establish the contribution of co-evolution. The method combines teacher-generated supervision, data verification, principle revision, and adaptive curricula, but does not adequately isolate their effects or those of additional resources. Improvements are difficult to attribute to individual principles, and the relationship between principle-selection criteria and subsequent learning gains remains unvalidated. The substantial degradation of the raw-data training baseline also lacks sufficient explanation.
-
Evaluation coverage and statistical reliability remain limited. The 2D VQA evaluation lacks expert-level, complex medical reasoning tasks, and independent repeated runs or confidence intervals are not clearly reported. On IU-Xray, the 8B model improves over the strongest baseline by only 0.92 RaTEScore points, while the 32B model performs worse than the 8B model. The stability of the gains and benefits of scaling therefore remain unclear.
Questions for authors
-
Under the same teachers, data sources, and comparable budgets, could you separately establish the contributions of principle evolution and adaptive curricula, including PPO’s advantage over simpler allocation strategies? Could you also explain the raw-data baseline’s degradation and validate the relationship between principle-selection criteria and subsequent learning gains?
-
How are patient-, study-, and image-level overlaps excluded across support data, internal validation sets, and external tests, particularly between PubMedVision-IU and IU-Xray? Was R7 selected exclusively using a predefined internal validation criterion?
-
How many independent runs underlie the main results, and could you provide standard deviations and paired confidence intervals? Additionally, why is CURE-4B absent from the IU-Xray comparison, and why does the 32B model underperform the 8B model on this dataset?
-
Why were complex reasoning benchmarks such as MedXpertQA-MM or the Health & Medicine portion of MMMU not included? How do the existing results support generalization to complex medical reasoning?
3 comments
Thank you for the responses and additional experiments. The new ablations and MMMU results, together with the clarifications on data separation and checkpoint selection, resolve most of my concerns. I am raising my rating to Weak Accept. I still note that sampling multiple times from a single checkpoint does not establish stability across independent training runs.
Thank you for the updated assessment
Thank you for reading our responses and for raising your rating. We are glad that the new ablations, the MMMU results, and the clarifications on data separation and checkpoint selection address most of your concerns. All of them are now in the revised PDF (App. C.2, C.4, C.6, D.4–D.7).
We agree on run-to-run stability. The three sampling seeds measure the decoding variability of one checkpoint, not the variability across training runs, and App. C.4 now states that each learner was trained once. The training-level evidence we have is that the pipeline, run independently on four learners from three model families, improves over each base in all 16 comparisons (Sec. 5.1; App. B).
Thank you again for the constructive review.
Thank you for the thorough questions: component ablations, overlap screening, paired confidence intervals, and MMMU
Q1 & W1 (Contributions, PPO, raw data, selection criteria). The paper already ablates both components with the same teachers, data, budget, and R5 checkpoint (Table 2, Sec. 5.3; App. D). We add a uniform split across capability cells as the simpler curriculum, to be added to the revision. Change from R5 on each benchmark:
Of the two, principle evolution has the larger effect, and PPO beats uniform allocation on all six with either principle set.
PPO also optimizes cumulative gain (Eq. 5), and its choices follow the learner's state. After localization regressed at R4 (60.27 → 56.16), it switched to repair-heavy for R5 and R6, and localization recovered to 59.36 and 62.10 (Fig. 6).
The raw-data baseline degrades because its sources (App. D.3) are not task-aligned (58.3% of 2D records are figure descriptions; Fig. 10), not verified, not targeted, and not adapted. CoEvoMed's agents handle each, and it reaches 91.60 on SLAKE (Table 9).
Principles chosen by our criteria do produce later learning gains. In our logs, 12 of 15 accepted revisions turned their target cell's net repair (validation cases fixed minus broken) positive in the next window, versus 6 of 20 eligible principles kept unrevised. By design (Sec. 4.3; App. B), a principle is replaced only when the validation cases it fixes no longer outnumber those it breaks, that is, when it stops helping the learner. A replacement is chosen only if a review confirms it targets the errors the learner still makes.
Q2 (Overlaps; R7). We excluded patient-, study-, and image-level overlaps by design: support, validation, and test data are separate partitions (Sec. 1; App. C.6), enforced in our code. We screened all 11 2D and 5 3D sources against every external test at four levels: IDs (including study, case, and PMC article IDs), image paths, pixel hashes (exact and near-duplicate), and normalized text. All checks found zero overlap, including PubMedVision-IU vs. IU-Xray.
R7 was selected on internal validation only: it has the highest equal-weight mean of QA and report scores, whose per-round change is the planner's reward (App. C.4). External benchmark scores never feed back into training or checkpoint selection (App. A, C.6).
Q3 & W2 (Runs and intervals; CURE-4B; 32B on IU-Xray). Each of four learners (Sec. 5.1; App. B) was trained once, and all 16 comparisons beat the base. Table 1 and Fig. 3 use greedy decoding (App. C.1); for the std, we re-evaluated CoEvoMed-8B (R7) by sampling with three seeds. Paired CIs compare the greedy run with Hulu-Med-7B, rerun with the same prompts and its own evaluation code (MIMIC-CXR: top baseline Qwen3.6-27B):
Sampled means stay within one std of the greedy scores, and every CI excludes zero. The rerun scores below Hulu-Med's published numbers, which Table 1 keeps, so its margins are conservative. On IU-Xray, the gain is 2.16 under the same protocol and 0.92 against the published 67.30.
CURE-4B is out of domain (trained on MIMIC-CXR-derived data), scoring 28.65 zero-shot on IU-Xray, below every model in Fig. 3.
IU-Xray references are short (30.5 words; MIMIC-CXR 53.4): 8B matches them (30.7), 32B writes longer ones (40.1).
Q4 & W2 (Complex reasoning). We chose benchmarks that span the data clinical reasoning works on, from 2D images to 3D volumes and from short answers to full reports:
On the MMMU Health & Medicine test set (1,752 questions, public answers, App. C.1 protocol), CoEvoMed-8B scores 62.04, versus 49.66 for its base and 61.30 for MediX-R1-8B, a medical model on the same base. We will add it to the revision.
The results support generalization: CoEvoMed ranks first on all nine benchmarks (Table 1; Figs. 3–4).