- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces PMC-InterCPT, a biomedical corpus for multimodal continued pretraining constructed from PMC articles. The pipeline recovers captions, associates figures with explicitly referenced body-text passages, repairs discontinuous contexts, and filters records for textual quality and medical relevance. It further adjusts the mixture of visual, quantitative, mechanistic, and auxiliary evidence. The resulting corpus contains 10.11M samples and 9.63B tokens. Validation includes experiments on three model configurations, nine medical and general/scientific benchmarks, token-matched controls, pipeline and mixture ablations, and a figure–context reference-breaking control.
Critical strengths
-
A substantial and potentially useful biomedical training resource. The pipeline addresses some of the problems in large-scale literature extraction, including missing captions, duplicated text, discontinuous contexts, and unsupported image–context associations. Constructing and documenting a 9.63B-token corpus could provide value to the community independently of architectural novelty.
-
The token-matched raw-data comparison, fixed SFT stage, cross-backbone experiments, and reference-breaking control are useful controlled validation. In particular, the reference-breaking experiment preserves the images, captions, source articles, and context length while disrupting figure–context correspondence, providing evidence that the correspondence matters.
Critical weaknesses
-
The incremental contribution over existing context-enriched biomedical data construction is insufficiently established. Beyond earlier work such as MedICaT and LLaVA-Med, recent resources have substantially advanced this direction. Open-PMC (MICCAI 2025) enriches image descriptions with summarized in-text references; BIOMEDICA (CVPR 2025) preserves figure-reference context and article-level metadata; and Open-PMC-18M (MICCAI 2026) combines subfigure extraction, subcaption alignment, and contextual summaries derived from inline references at large scale. Although these approaches differ from the proposed interleaved generative CPT setting, they provide closely related alternatives for exploiting article context. The paper acknowledges some of these resources, but its experiments primarily compare against raw caption pairs and naive interleaving. Consequently, the additional benefit of reference- and coherence-constrained reconstruction over existing context-enrichment strategies remains insufficiently demonstrated. Comparisons under the same backbone, token budget, and data-mixture controls would strengthen the contribution.
-
Medical gains are modest and heterogeneous, and their attribution remains unclear. On Qwen3.5-4B, the medical average improves from 62.37 with SFT alone to 63.93, but much of this improvement comes from PretexEval (+6.58 points), while both MMMU medical benchmarks decline. The controlled 1B-token ablation also complicates the interpretation: reconstruction and quality filtering reach a medical average of 62.35, below the caption-only baseline of 62.90; the final BVE-enhanced mixture reaches 63.28. Thus, the experiments demonstrate benefits from the combined pipeline more clearly than a medical-performance advantage specifically attributable to context reconstruction. An equally filtered and mixture-matched caption-only control is needed to separate these effects.
Questions for authors
See Weakness
5 comments
Response to Reviewer Co68 [2/2]
W2: How much of the medical gain comes from figure-linked context rather than the mixture or PretexEval?
Ans: Thank you for asking for a direct comparison with an equally filtered, mixture-matched caption-only control. We added this comparison at a 1B-token CPT budget to test the contribution of figure-linked body text.
The comparison with SFT alone measures the overall benefit of continued pretraining on the full PMC-InterCPT corpus. The medical average improves, and the effect varies across benchmarks. In the earlier 1B comparison, the reconstructed and quality-filtered Natural mixture (62.35) and raw BIOMEDICA caption pairs (62.90) differ in input format, filtering, and data distribution, so the score gap cannot be attributed to figure-linked context alone. We therefore compare the interleaved run with a caption-only control matched for source pool, filtering rules, evidence proportions, and token budget. With this control, the interleaved run scores higher on all five medical benchmarks.
For the caption-only control, we retain images and captions, remove the body-text context, and recompute token counts before resampling to the same 45/30/20/5 BVE/QTE/MSE/AUX token allocation. Both runs use about 1B multimodal CPT tokens, Qwen3.5-4B-Base, and the same SFT setting.
R3-Table 2: Matched caption-only and interleaved CPT scores (%). Δ is interleaved minus caption-only in percentage points.
| Benchmark | Caption-only | Interleaved | Δ (pp) |
|---|---|---|---|
| MMMU-Med-Test | 58.33 | 59.36 | +1.03 |
| MMMU-Pro-Med-10 | 40.56 | 41.96 | +1.40 |
| PMC-VQA-clean | 61.20 | 61.50 | +0.30 |
| OmniMedVQA | 85.70 | 86.19 | +0.49 |
| PretexEval | 66.69 | 67.37 | +0.68 |
| Medical average | 62.50 | 63.28 | +0.78 |
The medical average rises by 0.78 points. The mean gain is slightly larger at 0.81 points when we exclude PretexEval, calculated from the per-benchmark differences before rounding the mean. These gains support including reconstructed, figure-linked context in medical multimodal pretraining under a matched data mixture and token budget. The general/scientific average also rises by 0.66 points, from 62.12 to 62.78. CharXiv gains 3.00 points, and ChartQA drops by 2.20 points. Figure 4(b) reports all benchmark differences, and the appendix lists the full scores.
References.
[1] Subramanian et al. MedICaT: A Dataset of Medical Images, Captions, and Textual References. Findings of EMNLP 2020.
[2] Baghbanzadeh et al. Advancing Medical Representation Learning Through High-Quality Data. MICCAI 2025.
[3] Baghbanzadeh et al. Open-PMC-18M: A High-Fidelity Large Scale Medical Dataset for Multimodal Representation Learning. MICCAI 2026.
[4] Lozano et al. BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literature. CVPR 2025.
Thank you again for the thoughtful feedback. We hope the comparison with related corpora and the results of the matched caption-only control clarify what our results establish. In light of this new evidence, we would be grateful if you would reconsider your rating. We welcome any follow-up questions.
Dear Reviewer Co68,
As the discussion period is drawing to a close, we wanted to gently follow up to see whether our responses and additional experiments have addressed your concerns. Following your suggestions, we added:
- Comparisons with Open-PMC and Open-PMC-18M (R3-Table 1) under the same backbone, approximately 1B-token CPT budget, and SFT setting. PMC-InterCPT's BVE-enhanced mixture achieves higher medical and general/scientific averages than both context-enriched biomedical corpora, with gains of 2.32 and 0.53 points over Open-PMC, and 0.87 and 0.39 points over Open-PMC-18M, respectively.
- An equally filtered, mixture-matched caption-only control (R3-Table 2), matching the source pool, evidence allocation, training budget, and SFT setting. Interleaved training improves all five medical benchmarks, with an average gain of 0.78 points. Excluding PretexEval, the gain remains 0.81 points, supporting the benefit of figure-linked context under the matched controls.
We would be happy to clarify any remaining questions before the discussion closes. Thank you again for your time and constructive feedback!
Best regards,
The Authors
Response to Reviewer Co68 [1/2]
We sincerely thank the reviewer for the detailed and constructive feedback. We are glad that you found the 9.63B-token corpus a potentially useful resource and that you found the token-matched comparisons, fixed SFT stage, cross-backbone experiments, and reference-breaking control informative. Below, we clarify the distinction from related corpora and describe the additional controls needed to interpret the medical gains.
W1: What does reference-constrained reconstruction add over existing context-enriched corpora?
Ans: Thank you for pointing out the need to compare with prior biomedical corpora that use article context. We added 1B-token CPT comparisons with Open-PMC [2] and Open-PMC-18M [3] using Qwen3.5-4B-Base, the same SFT data, and the same evaluation protocol.
The revised manuscript presents the public-corpus comparison in Section 4.5 and Table 4.
MedICaT [1] provides inline references, and Open-PMC and Open-PMC-18M use them to enrich image–text pairs. PMC-InterCPT uses the original figure references to build interleaved training sequences that can retain multiple figures, their captions, and the passages that connect them. The new comparison uses approximately 1B multimodal tokens per corpus, with the released context-enriched text for Open-PMC and the released subfigure–text pairs for Open-PMC-18M. We include raw BIOMEDICA [4] as a caption-pair baseline and report both the Natural and BVE-enhanced allocations of PMC-InterCPT.
As shown in R3-Table 1, BVE-enhanced improves the medical and general/scientific averages by 2.32 and 0.53 points over Open-PMC and by 0.87 and 0.39 points over Open-PMC-18M. Open-PMC-18M scores higher than BVE-enhanced on PretexEval and ChartQA, and raw BIOMEDICA scores 62.90 on the medical average. Under the Natural allocation, PMC-InterCPT is close to Open-PMC-18M on the medical average (62.35 versus 62.41) and improves the general/scientific average by 1.19 points.
These comparisons evaluate the corpora as configured, including their different data mixtures. The reference-breaking experiment in Section 4.4 directly tests the role of the original figure–context links within PMC-InterCPT. Under the same images, captions, source articles, and token budget, breaking those links reduces the medical and general/scientific averages by 1.75 and 3.40 points, respectively.