acceptodds
Withdrawn from ICLR 2027

InfiMed2: A Generalist Medical Multimodal Foundation Model from Contextual Evidence and Stability-Aware Supervision

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recent medical multimodal models have benefited from larger corpora, broader modality coverage, and stronger reasoning-oriented training, yet effective data design across continued pretraining (CPT) and post-training remains challenging. Medical sources vary substantially in structure, granularity, and information density, and their utility shifts as training progresses from broad knowledge acquisition to late-stage consolidation. Meanwhile, post-training is often dominated by short-form visual question answering, providing limited supervision for informative and answer-consistent explanations. We introduce InfiMed2, a family of 4B and 27B generalist medical multimodal foundation models built around stage-aware data design. We curate a 55.68B-token corpus that combines broad clinical knowledge with context-rich biomedical visual evidence through source-specific processing. Our CPT pipeline first adapts the vision encoder, then builds broad medical knowledge, and finally transitions to an evidence-focused data mixture during learning-rate decay. For supervised fine-tuning (SFT), we regenerate visual question-answering responses using answer stability, answer-masked reconstruction, and correctness-constrained selection to produce more informative and answer-consistent supervision. The 4B model is further optimized with reinforcement learning with verifiable rewards (RLVR). Across five medical multimodal benchmarks, InfiMed2-4B achieves 66.73% mean accuracy after RLVR, surpassing the larger Qwen3.5-9B, while InfiMed2-27B reaches 73.72%, the highest among the evaluated open-weight models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
medical multimodal foundation models, continued pretraining, medical data curation, medical visual question answering, multimodal instruction tuning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_tWajOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces InfiMed2, a family of 4B and 27B generalist medical multimodal foundation models developed through stage-aware design. The authors construct a 55.68B-token corpus from medical e-books, biomedical literature, web documents, and general multimodal data. The training pipeline combines vision-encoder adaptation, continued pretraining (CPT), and evidence-focused learning-rate decay. For post-training, the authors propose stability-aware response regeneration to improve answer-consistent explanatory supervision, followed by reinforcement learning with verifiable rewards (RLVR) for the 4B model. Experiments across five medical multimodal benchmarks demonstrate competitive performance against general-purpose and domain-specific models.

Critical strengths
  1. Comprehensive medical data curation and training pipeline

The paper presents a systematic approach to organizing heterogeneous medical evidence across different training stages. Its source-specific processing, context-rich multimodal data construction, and stage-dependent mixtures provide a practical framework for developing medical foundation models.

  1. Broad Evaluation with Informative Ablations

The paper evaluates InfiMed2 across five medical multimodal benchmarks with both 4B and 27B models, demonstrating competitive performance and improvements over their respective backbones. More importantly, the matched SFT baselines, CPT comparisons, and response-target ablations provide useful insights into the effects of different training stages. Although some improvements are modest, these evaluations offer empirical support for the overall training pipeline.

Critical weaknesses
  1. Minor Contribution Of CPT

Although the paper motivates stage-aware CPT and evidence-focused learning-rate decay, their individual contributions are not sufficiently isolated. Table 2 compares different CPT token budgets, but all CPT variants retain the same vision-adaptation and decay-stage design. Moreover, the improvements over SFT-only baselines are relatively modest. Without controlled comparisons against static data mixtures or alternative stage configurations, it remains unclear which aspects of the proposed training strategy account for these gains.

  1. Insufficient Stability Validation

The proposed stability-aware regeneration improves answer consistency, but its effectiveness in producing more faithful and informative medical explanations remains insufficiently validated. The reported 94.10% answer-reconstruction accuracy primarily measures answer recoverability rather than explanation correctness or visual grounding. Furthermore, Table 3 shows only a 0.58% average improvement over randomly generated responses, with mixed results across individual benchmarks. More direct evaluation of explanation quality would strengthen the justification for stability-aware supervision.

Questions for authors
  1. Concerns on Stage-Aware CPT

Table 2 examines different CPT token budgets, but the benefit of the proposed evidence-focused decay mixture remains unclear. Could the authors compare it with the original CPT mixture under the same training budget and learning-rate schedule?

  1. Insufficient Gain of Stability-Aware Supervision

Section 3.2.2 combines answer stability, correctness-constrained filtering, and entropy-based selection to construct training responses. Given the limited gains over random selection in Table 3, could the authors further isolate the contribution of stability-aware selection, for example, by comparing against randomly selected answer-correct responses under matched training conditions?

Reviewer_94PuOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

InfiMed2 adapts Qwen3.5 (4B/27B) to the medical domain through staged continued pretraining on a curated 55.68B-token corpus, stability-aware regeneration of VQA explanations for SFT, and RLVR, surpassing larger open-weight baselines on five multiple-choice medical benchmarks.

Critical strengths
  1. All models are evaluated under a unified protocol with matched SFT-only controls at two scales, yielding reasonably controlled comparisons.

  2. The source-specific curation pipeline is documented in detail. The 4B and 27B models, if released, would constitute practical resources for the community.

  3. The paper candidly reports non-monotonic CPT effects and a trade-off between explanatory and answer-only targets, offering useful practical guidance.

Critical weaknesses
  1. The cleaning procedure and composition of PMC-VQA-clean are not described, so the quality and reliability of this evaluation subset cannot be assessed.

  2. Despite the generalist claim, evaluation is limited to multiple-choice benchmarks, omitting report generation, open-ended question answering, and other diverse downstream tasks.

  3. SFT data sources are insufficiently described, and no commitment to releasing the training data is made, raising substantial concerns about reproducibility.

  4. Disabling native thinking under reasoning prompts and using an unexplained direct-answer prompt for OmniMedVQA may disadvantage baselines, undermining the fairness of the comparison in Table 1.

Questions for authors
  1. The cleaning process of PMC-VQA-clean is not described in sufficient detail, making its specific composition difficult to determine. How was the quality of this subset evaluated?

  2. The authors claim that InfiMed2 is a generalist medical model, yet all selected benchmarks are in multiple-choice format. Evaluation on a broader range of tasks, such as report generation and open-ended question answering, would be necessary to fully validate the claimed generality.

  3. The sources of data used in the SFT stage are insufficiently described, and the authors do not commit to releasing the training datasets. This raises concerns about reproducibility. At a minimum, the authors should release the training data permitted under ethical and copyright constraints.

  4. For the Qwen3.5 models, would disabling native thinking while using a reasoning prompt affect performance? Why does OmniMedVQA use a direct-answer prompt? Neither choice is justified in the paper. Given these inconsistencies, is the comparison in Table 1 fair?

Reviewer_LEjHOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes InfiMed2, a family of 4B and 27B medical multimodal foundation models built upon Qwen3.5. Its core components include a multi-source medical continual pretraining corpus containing 55.68B tokens, visual encoder adaptation and a staged continual pretraining (CPT) recipe, and a stability-aware response regeneration pipeline for medical VQA. The post-training pipeline estimates answer stability through eight samples from the same teacher model. For high-stability samples, it performs answer-masked reconstruction and correctness-constrained filtering; for medium-stability samples, it invokes a stronger teacher model to regenerate responses; and it filters out low-stability samples. The 4B model additionally undergoes reinforcement learning with verifiable answer rewards.

Critical strengths
  1. The work covers the complete training pipeline, from multi-source corpus cleaning, visual encoder adaptation, and continual pretraining to supervised fine-tuning and RLVR. The two model sizes, 4B and 27B, also provide preliminary evidence of consistency across model scales.
  2. The authors re-evaluate all models using unified prompts, generation lengths, and answer extraction rules, and include an SFT-only baseline using the same Qwen3.5 backbone.
Critical weaknesses
  1. The paper presents “staged data design” as a major contribution, but does not validate its key components. Table 2 only compares the absence of CPT with overall checkpoints trained using different numbers of warmup–stable tokens. It lacks individual ablations of visual encoder adaptation, fixed-mixture baselines, data reweighting during the learning-rate decay stage, individual data sources, and general-domain replay. Multi-stage training of medical VLMs has already been explored in prior works such as GMAI-VL and Patho-R1. The current evidence cannot distinguish the contributions of the proposed training recipe from those of training scale, corpus size, or standard continual pretraining.
  2. The experimental support for stability-aware supervision is weak. Table 3 only compares three types of final training targets using 20k PMC-VQA samples under a lightweight SFT setting. The average improvement over randomly generated responses is only 0.58 percentage points, with slight performance degradation on MMMU-Pro and PMC-VQA. These results are insufficient to establish the necessity of each design choice in the complete pipeline.
  3. The claims of more informative, answer-consistent explanations are not sufficiently supported by the current evaluation. All five main evaluation metrics are based on exact-match accuracy for multiple-choice questions, without experiments on open-ended clinical question answering, explanation factuality and visual grounding, hallucination, or calibration.
Questions for authors
  1. What are the specific mechanisms through which each module of the proposed method operates? Is there corresponding experimental evidence to support them?
  2. Beyond exact-match accuracy on multiple-choice questions, is there more comprehensive experimental evidence from benchmark evaluations to support the paper's claim of producing more informative and answer-consistent explanations?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.