acceptodds
Under review as a conference paper at ICLR 2027

Beyond Captions: Context-Grounded Reconstruction for Biomedical Multimodal Continued Pretraining

Abstract

Biomedical figures are explained not by captions alone but by body-text passages that discuss them. Yet current multimodal corpora typically reduce figures to isolated image-caption pairs, discarding this crucial context. Existing pipelines either omit this context or append it without enforcing the figure references that support each attachment, which can create unsupported image-text attachments and incoherent discourse. We introduce context-grounded reconstruction, a source-grounded framework that converts PubMed Central Open Access (PMC-OA) records into referentially coherent interleaved sequences. It recovers captions and source text, attaches context only through article-native figure references, repairs non-contiguous context, and prunes unsupported images. Starting from these reconstructed sequences, PMC-InterCPT first filters records for text quality and medical relevance, then applies evidence-aware allocation to form a 9.63B-token corpus for continued pretraining (CPT) of generative medical MLLMs. With fixed supervised fine-tuning (SFT), PMC-InterCPT improves Qwen3.5-4B-Base by 1.46 medical-average points and 3.11 general/scientific-average points over a token-matched raw source control, and surpasses a 42% larger raw-data run. Gains transfer to Qwen3.5-2B-Base and LLaVA-OneVision-1.5-4B-Base. Controlled ablations show that context-grounded reconstruction, rather than simply appending article context, is central to useful biomedical multimodal CPT.

Primary area:
datasets and benchmarks
Keywords:
Biomedical Multimodal Learning, Multimodal Large Language Models, Continued Pretraining, Multimodal Data Curation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Co68OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PMC-InterCPT, a biomedical corpus for multimodal continued pretraining constructed from PMC articles. The pipeline recovers captions, associates figures with explicitly referenced body-text passages, repairs discontinuous contexts, and filters records for textual quality and medical relevance. It further adjusts the mixture of visual, quantitative, mechanistic, and auxiliary evidence. The resulting corpus contains 10.11M samples and 9.63B tokens. Validation includes experiments on three model configurations, nine medical and general/scientific benchmarks, token-matched controls, pipeline and mixture ablations, and a figure–context reference-breaking control.

Critical strengths
  1. A substantial and potentially useful biomedical training resource. The pipeline addresses some of the problems in large-scale literature extraction, including missing captions, duplicated text, discontinuous contexts, and unsupported image–context associations. Constructing and documenting a 9.63B-token corpus could provide value to the community independently of architectural novelty.

  2. The token-matched raw-data comparison, fixed SFT stage, cross-backbone experiments, and reference-breaking control are useful controlled validation. In particular, the reference-breaking experiment preserves the images, captions, source articles, and context length while disrupting figure–context correspondence, providing evidence that the correspondence matters.

Critical weaknesses
  1. The incremental contribution over existing context-enriched biomedical data construction is insufficiently established. Beyond earlier work such as MedICaT and LLaVA-Med, recent resources have substantially advanced this direction. Open-PMC (MICCAI 2025) enriches image descriptions with summarized in-text references; BIOMEDICA (CVPR 2025) preserves figure-reference context and article-level metadata; and Open-PMC-18M (MICCAI 2026) combines subfigure extraction, subcaption alignment, and contextual summaries derived from inline references at large scale. Although these approaches differ from the proposed interleaved generative CPT setting, they provide closely related alternatives for exploiting article context. The paper acknowledges some of these resources, but its experiments primarily compare against raw caption pairs and naive interleaving. Consequently, the additional benefit of reference- and coherence-constrained reconstruction over existing context-enrichment strategies remains insufficiently demonstrated. Comparisons under the same backbone, token budget, and data-mixture controls would strengthen the contribution.

  2. Medical gains are modest and heterogeneous, and their attribution remains unclear. On Qwen3.5-4B, the medical average improves from 62.37 with SFT alone to 63.93, but much of this improvement comes from PretexEval (+6.58 points), while both MMMU medical benchmarks decline. The controlled 1B-token ablation also complicates the interpretation: reconstruction and quality filtering reach a medical average of 62.35, below the caption-only baseline of 62.90; the final BVE-enhanced mixture reaches 63.28. Thus, the experiments demonstrate benefits from the combined pipeline more clearly than a medical-performance advantage specifically attributable to context reconstruction. An equally filtered and mixture-matched caption-only control is needed to separate these effects.

Questions for authors

See Weakness

5 comments
AuthorsOpenReview ↗

Response to Reviewer Co68 [1/2]

We sincerely thank the reviewer for the detailed and constructive feedback. We are glad that you found the 9.63B-token corpus a potentially useful resource and that you found the token-matched comparisons, fixed SFT stage, cross-backbone experiments, and reference-breaking control informative. Below, we clarify the distinction from related corpora and describe the additional controls needed to interpret the medical gains.


W1: What does reference-constrained reconstruction add over existing context-enriched corpora?

Ans: Thank you for pointing out the need to compare with prior biomedical corpora that use article context. We added 1B-token CPT comparisons with Open-PMC [2] and Open-PMC-18M [3] using Qwen3.5-4B-Base, the same SFT data, and the same evaluation protocol.

The revised manuscript presents the public-corpus comparison in Section 4.5 and Table 4.

MedICaT [1] provides inline references, and Open-PMC and Open-PMC-18M use them to enrich image–text pairs. PMC-InterCPT uses the original figure references to build interleaved training sequences that can retain multiple figures, their captions, and the passages that connect them. The new comparison uses approximately 1B multimodal tokens per corpus, with the released context-enriched text for Open-PMC and the released subfigure–text pairs for Open-PMC-18M. We include raw BIOMEDICA [4] as a caption-pair baseline and report both the Natural and BVE-enhanced allocations of PMC-InterCPT.

R3-Table 1: Public-corpus and PMC-InterCPT comparison at approximately 1B CPT tokens. Averages (%) cover five medical and four general/scientific benchmarks, respectively.

CPT dataMedical avg.General/scientific avg.
Raw BIOMEDICA62.9061.41
Open-PMC60.9662.25
Open-PMC-18M62.4162.39
PMC-InterCPT (Natural)62.3563.58
PMC-InterCPT (BVE-enhanced)63.2862.78

As shown in R3-Table 1, BVE-enhanced improves the medical and general/scientific averages by 2.32 and 0.53 points over Open-PMC and by 0.87 and 0.39 points over Open-PMC-18M. Open-PMC-18M scores higher than BVE-enhanced on PretexEval and ChartQA, and raw BIOMEDICA scores 62.90 on the medical average. Under the Natural allocation, PMC-InterCPT is close to Open-PMC-18M on the medical average (62.35 versus 62.41) and improves the general/scientific average by 1.19 points.

These comparisons evaluate the corpora as configured, including their different data mixtures. The reference-breaking experiment in Section 4.4 directly tests the role of the original figure–context links within PMC-InterCPT. Under the same images, captions, source articles, and token budget, breaking those links reduces the medical and general/scientific averages by 1.75 and 3.40 points, respectively.

AuthorsOpenReview ↗

Response to Reviewer Co68 [2/2]

W2: How much of the medical gain comes from figure-linked context rather than the mixture or PretexEval?

Ans: Thank you for asking for a direct comparison with an equally filtered, mixture-matched caption-only control. We added this comparison at a 1B-token CPT budget to test the contribution of figure-linked body text.

The comparison with SFT alone measures the overall benefit of continued pretraining on the full PMC-InterCPT corpus. The medical average improves, and the effect varies across benchmarks. In the earlier 1B comparison, the reconstructed and quality-filtered Natural mixture (62.35) and raw BIOMEDICA caption pairs (62.90) differ in input format, filtering, and data distribution, so the score gap cannot be attributed to figure-linked context alone. We therefore compare the interleaved run with a caption-only control matched for source pool, filtering rules, evidence proportions, and token budget. With this control, the interleaved run scores higher on all five medical benchmarks.

For the caption-only control, we retain images and captions, remove the body-text context, and recompute token counts before resampling to the same 45/30/20/5 BVE/QTE/MSE/AUX token allocation. Both runs use about 1B multimodal CPT tokens, Qwen3.5-4B-Base, and the same SFT setting.

R3-Table 2: Matched caption-only and interleaved CPT scores (%). Δ is interleaved minus caption-only in percentage points.

BenchmarkCaption-onlyInterleavedΔ (pp)
MMMU-Med-Test58.3359.36+1.03
MMMU-Pro-Med-1040.5641.96+1.40
PMC-VQA-clean61.2061.50+0.30
OmniMedVQA85.7086.19+0.49
PretexEval66.6967.37+0.68
Medical average62.5063.28+0.78

The medical average rises by 0.78 points. The mean gain is slightly larger at 0.81 points when we exclude PretexEval, calculated from the per-benchmark differences before rounding the mean. These gains support including reconstructed, figure-linked context in medical multimodal pretraining under a matched data mixture and token budget. The general/scientific average also rises by 0.66 points, from 62.12 to 62.78. CharXiv gains 3.00 points, and ChartQA drops by 2.20 points. Figure 4(b) reports all benchmark differences, and the appendix lists the full scores.


References.

[1] Subramanian et al. MedICaT: A Dataset of Medical Images, Captions, and Textual References. Findings of EMNLP 2020.

[2] Baghbanzadeh et al. Advancing Medical Representation Learning Through High-Quality Data. MICCAI 2025.

[3] Baghbanzadeh et al. Open-PMC-18M: A High-Fidelity Large Scale Medical Dataset for Multimodal Representation Learning. MICCAI 2026.

[4] Lozano et al. BIOMEDICA: An Open Biomedical Image-Caption Archive, Dataset, and Vision-Language Models Derived from Scientific Literature. CVPR 2025.


Thank you again for the thoughtful feedback. We hope the comparison with related corpora and the results of the matched caption-only control clarify what our results establish. In light of this new evidence, we would be grateful if you would reconsider your rating. We welcome any follow-up questions.

Dear Reviewer Co68,

As the discussion period is drawing to a close, we wanted to gently follow up to see whether our responses and additional experiments have addressed your concerns. Following your suggestions, we added:

  • Comparisons with Open-PMC and Open-PMC-18M (R3-Table 1) under the same backbone, approximately 1B-token CPT budget, and SFT setting. PMC-InterCPT's BVE-enhanced mixture achieves higher medical and general/scientific averages than both context-enriched biomedical corpora, with gains of 2.32 and 0.53 points over Open-PMC, and 0.87 and 0.39 points over Open-PMC-18M, respectively.
  • An equally filtered, mixture-matched caption-only control (R3-Table 2), matching the source pool, evidence allocation, training budget, and SFT setting. Interleaved training improves all five medical benchmarks, with an average gain of 0.78 points. Excluding PretexEval, the gain remains 0.81 points, supporting the benefit of figure-linked context under the matched controls.

We would be happy to clarify any remaining questions before the discussion closes. Thank you again for your time and constructive feedback!

Best regards,
The Authors

Reviewer_g6AsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PMC-InterCPT, a 9.63B-token biomedical multimodal CPT corpus constructed from PMC-OA. The proposed pipeline combines context-grounded reconstruction, quality filtering, and evidence-aware allocation to preserve figure-referenced article context. Experiments across multiple MLLM backbones demonstrate improvements over raw biomedical pretraining data, supported by ablation studies and counterfactual evaluations.

Critical strengths
  1. The paper presents a systematic pipeline for reconstructing biomedical interleaved image-text data from scientific articles. Preserving figure-referenced context addresses a meaningful limitation of conventional image-caption pretraining corpora.

  2. The paper includes token-matched comparisons, component ablations, reference-breaking experiments, and cross-architecture evaluations. These experiments provide convincing evidence that figure-linked context contributes to downstream performance.

Critical weaknesses
  1. Limited direct validation of reconstructed figure-context alignment. The framework relies on article-native figure references to establish image-context associations. However, no systematic evaluation of the correctness and clinical relevance of these reconstructed relationships is provided.

  2. Limited evidence for the specific benefits of multi-figure interleaving. Although the proposed corpus preserves multi-figure relationships, the downstream evaluations mainly assess general medical VQA and scientific understanding. It remains unclear whether the method improves reasoning across multiple figures or long-range article context.

Questions for authors
  1. Can the authors provide a quantitative or human evaluation of figure-context alignment accuracy in the reconstructed corpus?

  2. Can the authors evaluate the proposed corpus on tasks requiring explicit multi-figure reasoning or cross-figure evidence integration?

  3. Can the authors report results across multiple random seeds or provide confidence intervals to establish the reliability of the observed improvements?

  4. Is benchmark decontamination performed beyond PMC-VQA, particularly for scientific figure understanding benchmarks that may overlap with PMC articles?

  5. Figure 1 appears somewhat over-designed, with excessive decorative elements, dense annotations, and small text. Can you enhance the figure?

10 comments
AuthorsOpenReview ↗

Response to Reviewer g6As [1/3]

We sincerely thank the reviewer for the detailed and constructive feedback. We are glad that you found preserving figure-linked context a meaningful improvement over conventional image-caption data, and that the token-matched comparisons, ablations, reference-breaking control, and cross-architecture evaluations helped establish its value. Below, we provide point-by-point clarifications, additional evaluation results, and details on the remaining validation.


W1&Q1: Direct validation of figure-context alignment and clinical relevance.

Ans: Thank you for asking for direct evidence of alignment and clinical relevance. We audited 200 reconstructed records with retained article body text (100 single-figure and 100 multi-figure records; 327 figures). One experienced annotator labeled these records for alignment and clinical usefulness.

Figure-context alignment. We counted a record as fully aligned only if the retained body text specifically discussed every included figure, either as a whole or through one of its panels. A figure-number mention or a matching caption alone was insufficient. 199/200 records (99.5%) were fully aligned: 100/100 single-figure and 99/100 multi-figure records. At the figure level, 326/327 (99.7%) were aligned. In the only partially aligned record, the retained paragraph discussed gene expression patterns in the neocortex and hippocampus shown in Fig. 5, but did not discuss the expression heatmap or gene-specific panels in Fig. 3, even though both figures came from the same article.

Clinical usefulness. We first screened the aligned figures for clinical relevance, asking whether each depicted a clinical finding. For figures that passed this screen, we then checked whether the retained body text provided information beyond the caption that helped interpret the finding, such as how it was confirmed or how it changed on follow-up. The screen identified 111 clinically relevant figures, and the context added such information for all 111 in the audited sample.

Together, these results show high figure-context alignment in the audited subset of the reconstructed corpus and demonstrate that the retained context provides clinically useful information beyond the captions.

W2&Q2: Does multi-figure interleaving improve cross-figure reasoning?

Ans: Thank you for asking for a direct evaluation of reasoning across figures. We added four multi-image benchmarks to the evaluation of the full-scale Qwen3.5-4B checkpoints. MedFrameQA [3] and MedThinkVQA [4] test medical reasoning across images, MuirBench [1] tests general multi-image understanding, and MultiChartQA [5] tests reasoning across charts.

The revised manuscript presents this evaluation in Section 4.7 and Table 6.

R2-Table 3: Multi-image benchmark scores (%). Overall is the unweighted mean across the four benchmarks.

CPT dataMedFrameQAMedThinkVQAMuirBenchMultiChartQAOverall
None (SFT only)57.6341.1156.9645.8350.38
BIOMEDICA (13.65B)58.7540.5557.1543.9550.10
BIOMEDICA (9.63B)58.6140.2857.1946.5550.66
PMC-InterCPT (9.63B)59.3543.0658.2746.0651.69

At the same 9.63B-token CPT budget, PMC-InterCPT improves the overall average by 1.03 percentage points over raw BIOMEDICA. The gains cover both medical benchmarks (+0.74 on MedFrameQA and +2.78 on MedThinkVQA) and MuirBench (+1.08). On MultiChartQA, the token-matched BIOMEDICA baseline leads by 0.49 points. PMC-InterCPT also improves over SFT alone on all four benchmarks. These results give direct downstream evidence for the full corpus on tasks involving multiple images.

AuthorsOpenReview ↗

Response to Reviewer g6As [2/3]

Q3: How reliable are the full-corpus gains and the choice of the BVE-enhanced mixture?

Ans: Thank you for asking us to test the reliability of the reported gains. We address this in two ways. First, we use 95% confidence intervals to assess whether the gains from the full PMC-InterCPT corpus hold beyond evaluation-sample variation at the 4B and 2B scales. Second, we rerun both CPT and SFT for the Natural and BVE-enhanced mixtures at 1B tokens to test whether the medical advantage of the selected mixture holds across training seeds.

1. Full-corpus gains at fixed checkpoints. For each model scale, we report the difference between PMC-InterCPT and three baselines: SFT only, full raw BIOMEDICA, and token-matched raw BIOMEDICA. The intervals quantify evaluation uncertainty for the five-benchmark medical average and the four-benchmark general/scientific average at fixed checkpoints.

R2-Table 1: PMC-InterCPT minus each baseline, in percentage points, with 95% confidence intervals.

ScaleBaselineMedical Δ [95% CI]General/scientific Δ [95% CI]
4BSFT only+1.56 [+0.54, +2.56]+0.07 [-0.87, +1.02]
4BFull raw BIOMEDICA+1.58 [+0.65, +2.52]+2.44 [+1.54, +3.35]
4BToken-matched raw BIOMEDICA+1.46 [+0.43, +2.32]+3.11 [+2.11, +4.11]
2BSFT only+0.99 [-0.16, +1.85]+0.45 [-0.55, +1.43]
2BFull raw BIOMEDICA+1.53 [+0.52, +2.51]+2.34 [+1.34, +3.27]
2BToken-matched raw BIOMEDICA+2.36 [+1.14, +3.17]+2.88 [+1.57, +3.63]

PMC-InterCPT improves both averages at both model scales, even when the raw BIOMEDICA baseline uses the same token budget. All eight intervals against the two raw-data baselines are above zero, including when the baseline uses the larger, full raw corpus. This supports the value of corpus construction beyond increasing the volume of raw training data. Against SFT only, the 4B medical gain also excludes zero; the other three intervals include zero.

2. Mixture choice across training seeds. We ran the full CPT+SFT pipeline for both 1B mixtures under three matched seed groups, varying the seed in both stages and keeping the token budget fixed.

R2-Table 2: Natural and BVE-enhanced across three matched CPT+SFT seed groups. Averages (%) cover five medical and four general/scientific benchmarks, respectively. Δ is BVE-enhanced minus Natural in percentage points before rounding. SD is the sample standard deviation across seed groups.

Seed groupMedical: NaturalMedical: BVE-enhancedMedical ΔGen./Sci.: NaturalGen./Sci.: BVE-enhancedGen./Sci. Δ
G162.3563.28+0.9263.5862.78-0.80
G261.2462.55+1.3262.6962.86+0.17
G361.9862.77+0.7962.5663.00+0.44
Mean ± SD61.85 ± 0.5762.87 ± 0.37+1.01 ± 0.2762.94 ± 0.5662.88 ± 0.11-0.06 ± 0.65

BVE-enhanced improves the medical average in all three matched seed groups, by 0.79 to 1.32 points. The general/scientific difference changes sign and averages close to zero. This supports choosing BVE-enhanced over Natural when prioritizing medical performance, without implying that the selected ratio is optimal. Appendix F.1 of the revised manuscript reports the confidence intervals in Table 18 and the per-seed results in Table 19.

AuthorsOpenReview ↗

Response to Reviewer g6As [3/3]

Q4: Was benchmark overlap checked beyond PMC-VQA?

Ans: Thank you for asking about overlap with scientific figure benchmarks. We extended the overlap check to all visual benchmarks in our main evaluation suite. We compared their evaluation images with all images in the final PMC-InterCPT CPT WebDataset (WDS) and found zero image matches, including for CharXiv-Val and SciVQA-Test.

For these two scientific figure benchmarks, we also retrieved the source papers from arXiv and compared both the papers and their figures with the source papers and images in the final CPT corpus. We found no overlapping papers or matching figures.

Before CPT, we applied article-level decontamination for PMC-VQA-clean [2]. We excluded all records from the 1,145 PMC articles represented in its evaluation split, removing 4,363 records by source article ID. The revised manuscript describes the article-level exclusion and the overlap checks.

Q5: Improve the readability of Figure 1.

Ans: Thank you for pointing out that the dense annotations and small labels made Figure 1 difficult to read. We redrew Figure 1 in the revised manuscript with more prominent panel headings, shorter labels and examples, and fewer secondary annotations. The input and output panels show the same two biomedical figures and captions, making it easier to see what conventional caption pairs omit and what figure-linked context PMC-InterCPT restores.


References.

[1] Wang et al. MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding. ICLR 2025.

[2] Zhang et al. PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering. 2023.

[3] Yu et al. MedFrameQA: A Multi-Image Medical VQA Benchmark for Clinical Reasoning. EMNLP 2026.

[4] Yao et al. Medical Thinking with Multiple Images. 2026.

[5] Zhu et al. MultiChartQA: Benchmarking Vision-Language Models on Multi-Chart Problems. NAACL 2025.


Thank you again for the thoughtful feedback. We hope the alignment audit results, uncertainty analysis, overlap checks, and figure changes address your concerns. In light of this new evidence, we would be grateful if you would reconsider your rating. We welcome any follow-up questions.

Reviewer_K5vROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes to continue to pretrain VLMs on medical-centric interleaved multimodal sequences.The author construct PMC-InterCPT corup from data collected from PubMed Central Open Access. Experiment results show that CPT on proposed corpus leads to substantial gain across diverse benchmarks

Critical strengths
  1. The authors commit to opensource the 9.63B-token corupus if paper is accepted. This is a valuable artifact and can have profound impact to building strong medical foundation models.
  2. Empirical results show that CPT on PMC-InterCPT achieves considerable gains across model sizes and benchmarks.
  3. The author conducted detailed ablation studies on various design choices of dataset construct, which can inspire future works on pretraining.
Critical weaknesses
  1. The relevance classifier report 93.4% accuracy, suggesting the dataset contains 6.6% noise that is not relevant. While some noise is expected for large scale corpus, this error rates warrants further investigation on what are the failure cases and whether it would introduce systematic bias. (For example, the classifier may completely fail on certain low-frequency medical subjects by classifying related documents as irrelevant) A recall centric analysis could further strengthen the soundness of classifier.
  2. Gemini-3.1-pro-preview is used as the teacher to provide source labels of medical relevance, however it has not been established such annotation itself is of high quality and has low hallucination. The author suggest "VLM-based image-text relevance" are not used because of hallucination, but it is not discussed how is hallucination measured on if text-only approach truly offers a lower rate. Ideally, this should be conducted by sampled audit with real human annotators.
Questions for authors

See weakness Additional question

  1. The author uses evidence-based resampling to balance image classes, however BVE-enhanced contains only 45% BVE, leaving considerable design space between BVE-enhanced vs BVE-only (100%). The author should discuss how is the ratio chosen, is it purely heuristic based? Will enhancing BVE beyond 45% further improve the performance? I understand that for such large scale corpus, it may be expensive to exhaustively search for optimal ratios, but some discussions and analysis of the correlation between the frequency of BVE data and benchmark performance would further strengthen the value of this paper and benefits future works in this domain
4 comments
AuthorsOpenReview ↗

Response to Reviewer K5vR [1/2]

We sincerely thank the reviewer for the detailed and constructive feedback. We are glad that you found the planned release of the 9.63B-token corpus valuable, the gains across model sizes and benchmarks encouraging, and the construction ablations informative. Below, we respond point by point with additional analyses, experimental details, and clarifications.


W1: Medical relevance recall and the risk of missing low-frequency medical topics.

Ans: Thank you for raising the possibility that the classifier misses records on low-frequency medical topics. We compared the recall of our medical-relevance classifier on low-frequency and common topics. We grouped topics by the number of pre-filter PMC articles associated with each Medical Subject Headings (MeSH) descriptor.

For each article, we paired a predicted medical input with evidence of the target MeSH topic and a predicted non-medical input. The low-frequency and common groups each contained 200 inputs, with 100 classified as medical and 100 as non-medical by our medical-relevance classifier. For the inputs we reviewed manually, we used human annotations as the medical-relevance reference labels.

In the paired sample, the classifier recalled 99/113 (87.61%) medically relevant inputs in the low-frequency group and 94/110 (85.45%) in the common group. Recall was similar for low-frequency and common topics in the selected article pairs. Appendix C.5 and Figure 5 report the sampling design and results.

The 6.6% is the error rate on the balanced held-out set with Gemini reference labels (19/286, including 12 false negatives, 5 false positives, and 2 invalid outputs). Medical recall on that set was 90.2% (129/143) and precision was 96.3% (129/134). We added these results to Appendix C.4 of the revised manuscript (Table 16). Inputs classified as non-medical can still be used for training through the high-quality general pool if their quality score is 2. This reduces the risk of discarding useful training data due to false-negative relevance predictions.

W2: Reliability of the Gemini teacher and the rationale for omitting a hard image-text relevance filter.

Ans: Thank you for asking us to evaluate the teacher and explain why we did not use VLM scores as a hard filter.

1. Failure cases of image-conditioned scoring. Appendix C.6 shows two examples with their images: (a) the model cited an image tag that was absent from the input as a reason to lower the quality score; (b) it failed to detect a genuine mismatch, asserting that the text matched the figure even though panel B shows HT29/SW480 and the caption and passage name SW620/HCT116.

2. Why we use text-only annotation. Our records contain one or more original PMC figure-caption pairs followed by article context. Reference-constrained interleaving uses explicit figure references to attach the context to the corresponding figures and captions. This preserves the source associations without relying on a VLM to infer them, although discrepancies within the source material can remain. We therefore focus quality annotation on an aspect the references cannot establish: whether the retained text is usable for pretraining.

We compared the two workflows on 200 reconstructed records. The text-only workflow scored captions and passages; the image-conditioned workflow also saw the figures and could penalize mismatches. In all 67 quality-score disagreements, the image-conditioned workflow assigned 1. In the 29 changes from 0 to 1, it often treated LaTeX, SVG, or malformed table artifacts as tolerable because some prose was still readable. This can let corrupted text through the filter and weaken the training signal. In the 38 changes from 2 to 1, some rationales described the input as clear and aligned without naming a defect. Such downgrades can move otherwise eligible records into excluded pools, reducing the amount of usable data available for evidence-aware allocation.

Against independent text-quality reference scores manually annotated, exact agreement was 87.5% (175/200) for the text-only workflow and 71.0% (142/200) for the image-conditioned workflow.

Text-only inference also avoids image encoding and the processing of image tokens for both medical-relevance and quality classification, reducing computation across millions of records. Therefore, we use text-only annotation.

AuthorsOpenReview ↗

Response to Reviewer K5vR [2/2]

Q1: Why choose 45% BVE, and does further enrichment improve performance?

Ans: Thank you for suggesting that we test a higher BVE share. We added a 70% BVE mixture to assess whether increasing the share beyond 45% improves benchmark performance.

In the revised manuscript, Section 3.4 describes the allocation strategy, and Section 4.3 reports the mixture comparison in Table 3 and Figure 3(b).

We compare the five mixtures using the same Qwen3.5-4B-Base backbone, 1B-token CPT budget, SFT data, and training hyperparameters. For the new mixture, we allocate 70% of the tokens to BVE and split the remaining 30% among QTE, MSE, and AUX in the same 6:4:1 ratio as in BVE-enhanced. The resulting target shares are 70%, 16.36%, 10.91%, and 2.73%, respectively.

R1-Table 2: Selected scores and averages (%) for the five 1B-token mixtures.

ConfigurationTarget BVE shareMedical avg.General/scientific avg.OmniMedVQAChartQAPretexEvalPMC-VQA-clean
Natural28.13%62.3563.5886.3877.0862.9261.10
Balanced33%62.4663.4385.5676.9264.6562.80
BVE-enhanced45%63.2862.7886.1976.6467.3761.50
BVE-enriched70%62.5462.6386.6478.1264.7860.10
BVE-only100%62.6762.8486.5177.2065.7661.75

As shown in R1-Table 2, increasing the BVE share from 45% to 70% lowered the medical average from 63.28 to 62.54 and the general/scientific average from 62.78 to 62.63. OmniMedVQA and ChartQA improved by 0.45 and 1.48 percentage points, while PretexEval and PMC-VQA-clean fell by 2.59 and 1.40 points.

We selected 45% BVE for the final corpus because it achieved the highest medical average and supported a larger, more diverse corpus. BVE makes up 28.13% of the filtered source pool (4.33B tokens). We avoid repeating records to reach a target share because repeated records consume CPT tokens without adding new evidence. The available BVE limits a 70% mixture to roughly 6.2B tokens. At 45% BVE, we can build the 9.63B-token corpus and reserve 55% of its tokens for quantitative, mechanistic, and auxiliary evidence.


Thank you again for the thoughtful feedback. We hope the medical-relevance audit, the clarification of the teacher's role, and the mixture comparisons address your concerns. In light of this new evidence, we would be grateful if you would reconsider your rating. We welcome any follow-up questions.

Thanks the authors for responses. my concerns have been addressed.

AuthorsOpenReview ↗

Response to Reviewer K5vR

We sincerely appreciate your thoughtful feedback and are glad to hear that our responses have addressed your concerns. We will incorporate the additional analyses, experimental results, and clarifications into the revised manuscript. Thank you again for your time and constructive suggestions throughout the review process. If you have any further questions or concerns, we would be happy to provide additional clarification.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair,

Thank you for your time and effort in overseeing our submission. We summarize the discussion status and the additional evidence provided during rebuttal to facilitate your assessment.

Our work introduces PMC-InterCPT, a 9.63B-token biomedical interleaved corpus that preserves figure-linked article context for multimodal continued pretraining. The reviewers’ main concerns concerned the reliability of data curation, validation of reconstructed relationships, cross-image reasoning, comparisons with existing biomedical corpora, and attribution of the observed gains.

Following our responses:

  • Reviewer K5vR: We provided recall-focused analyses of low-frequency medical topics, a paired evaluation of annotation workflows, and an additional 70% BVE experiment to examine the selected data mixture.
  • Reviewer g6As: After considering the alignment audit, additional multi-image evaluations, uncertainty analyses, and further cross-image clarifications, the reviewer concluded: “It has addressed most of my concerns.”
  • Reviewer Co68: We provided comparisons with Open-PMC and Open-PMC-18M and an equally filtered, mixture-matched caption-only control to address the concerns about related corpora and the contribution of reconstructed context.

The main additions during rebuttal are:

An equally filtered, mixture-matched caption-only control.

Using the same filtered source pool, 45/30/20/5 evidence allocation, approximately 1B CPT tokens, backbone, and SFT setting, interleaved training improves all five medical benchmarks over caption-only training. The medical average rises from 62.50 to 63.28, and the general/scientific average improves by 0.66 points, supporting the value of figure-linked body-text context.

Comparisons with existing context-enriched biomedical corpora.

We added approximately 1B-token comparisons with Open-PMC and Open-PMC-18M under the same backbone and SFT setting. PMC-InterCPT improves the medical and general/scientific averages by 2.32 and 0.53 points over Open-PMC, and 0.87 and 0.39 points over Open-PMC-18M, respectively.

Mixture sensitivity and uncertainty analyses.

The new 70% BVE experiment extends the comparison to five allocations. The selected 45% mixture retains the highest medical average (63.28 versus 62.54 at 70%) while supporting greater corpus size and evidence diversity without repeating records. Across three matched seed groups, it consistently outperforms Natural on the medical average, with a mean gain of 1.01 points. We also added confidence intervals for full-corpus gains at the 4B and 2B scales.

Direct human validation of figure–context alignment and clinical usefulness.

An experienced annotator audited 200 records containing 327 figures. 199/200 records (99.5%) were fully aligned, requiring the body text to discuss every included figure. For all 111 clinically relevant figures identified in this sample, the retained context provided useful information beyond the captions.

Expanded multi-image evaluation and cross-image analyses.

We added MedFrameQA, MedThinkVQA, MuirBench, and MultiChartQA evaluations. Follow-up analyses distinguish single-chart questions from comparative and sequential cross-chart reasoning, examine six relational or sequential MuirBench tasks, and assess automatically identified medical subsets supported by official references. These analyses clarify the evidence for cross-image integration beyond aggregate multi-image scores.

Recall-focused curation analysis and paired teacher-workflow evaluation.

In the paired MeSH-frequency audit, medical-relevance recall was 87.61% for low-frequency topics and 85.45% for common topics. We clarified that 6.6% denotes held-out classification error, rather than irrelevant content in the final corpus. A 200-record paired comparison further examines text-only versus image-conditioned scoring, filtering consequences, and illustrative scoring failures.

Expanded benchmark-overlap checks and clearer presentation.

We found no matching images between the final CPT corpus and the main visual evaluation benchmarks. Additional source-paper and figure comparisons found no overlap for CharXiv-Val and SciVQA-Test, complementing training-time article-level exclusion for PMC-VQA-clean. We also redrew Figure 1 and clarified the manuscript.

We remain committed to publicly releasing the full 9.63B-token PMC-InterCPT dataset to support reproducible biomedical multimodal research.

We respectfully ask that these additional findings and the reviewers’ follow-up assessments be considered alongside the original submission. Thank you again for your careful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.