acceptodds
Under review as a conference paper at ICLR 2027

Ophth-500K: Curating Unstructured Video Streams for Scaling Multimodal Ophthalmology Foundation Models

Abstract

Large-scale vision-language learning in ophthalmology remains constrained by the limited availability of multimodal data with clinically descriptive supervision. We introduce Ophth-500K, a large-scale multimodal ophthalmology dataset constructed from 14,769 hours of expert-led videos. Our automated data engine identifies clinically relevant ophthalmic content, aligns visual segments with expert narration, and converts continuous video streams into structured image, text, and audio samples. Ophth-500K contains 467,697 aligned multimodal samples spanning Optical Coherence Tomography, Color Fundus Photography, and Ultra Widefield imaging. Its source content covers 116 languages and more than 100 countries and regions, while UMLS-based analysis identifies 1,885 anatomical concepts, 689 pathologic findings, and 3,379 disease concepts. To evaluate the learning value of the curated supervision, we train OphthCLIP using a standard contrastive vision-language objective. OphthCLIP substantially improves cross-modal retrieval over general-purpose, medical, and ophthalmology-specific baselines, achieving 32.6% Recall@1 for image-to-text retrieval and 33.2% for text-to-image retrieval. These results demonstrate the value of expert ophthalmology videos as a scalable source of structured multimodal supervision.

Primary area:
datasets and benchmarks
Keywords:
Medical Multimodal Datasets, Data Curation, Ophthalmic Imaging, Vision-Language Foundation Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fdQeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents Ophth-500K, a dataset of 467,697 image–text–audio samples curated from 29,465 publicly accessible ophthalmology videos totaling 14,769 hours. The pipeline screens video metadata, classifies sampled frames for ophthalmic relevance, segments temporally stable visual content, aligns nearby speech using Whisper timestamps, extracts ophthalmic regions using SAM3 and BiomedCLIP, masks detected identifying information, and uses Qwen3-VL-8B to convert the image and transcript into a concise clinical description.

Critical strengths

Ophth-500K is substantially larger than most public ophthalmology image–text resources and retains audio in addition to images and text. The pipeline tackles clinically relevant challenges including temporal narration offset, panel extraction, and privacy filtering.

Critical weaknesses
  1. Dataset quality and clinical validity are insufficiently evaluated. The paper reports no clinician audit or measurements of alignment, caption factuality, modality classification, privacy filtering, or cross-video duplication. Retrieval on captions produced by the same pipeline mainly demonstrates in-domain learnability rather than clinical accuracy or downstream utility.

  2. The release, licensing, and privacy basis is unclear. Publicly accessible videos are not necessarily redistributable. The paper does not specify applicable licenses, releasable content, platform compliance, takedown procedures, or privacy-audit results, despite labeling the dataset as public.

Questions for authors
  1. Are all 467,697 Qwen-generated descriptions reviewed by humans, as the AI Use Statement may imply?

  2. Do the authors have evaluation on an independently curated ophthalmology dataset or a downstream clinical task that does not use captions generated by the same pipeline?

1 comment
AuthorsOpenReview ↗

Clinical validity and external evaluation

We thank the reviewer for the constructive feedback.

W1/Q1

The original expert verification referred to manual inspection of selected samples, not exhaustive clinical review of all 467,697 descriptions. We subsequently conducted a systematic audit of 100 training and 100 evaluation samples across seven quality criteria. Three independent medical imaging researchers assessed the samples, with disagreements adjudicated by a senior clinician. The audit includes image validity, modality, cropping, image–text alignment, invented content, clinical accuracy, and patient privacy, together with inter-rater agreement statistics. The complete results and protocol are reported in our response to Reviewer bjw1 (W1/Q1).

W2

Ophth-500K has not yet been publicly released. We plan a two-stage release strategy, subject to applicable source licenses, platform policies, institutional requirements, and relevant IRB requirements.

First, we plan to release the complete data curation pipeline, source video identifiers, temporal annotations, processing configurations, and pretrained OphthCLIP weights through GitHub and Hugging Face. Researchers can reconstruct the dataset from its original sources where access and permissions allow.

Second, we plan to progressively release curated image–text pairs following individual manual review for clinical content, privacy, and redistribution eligibility. We target approximately 50,000 samples per month, subject to the necessary reviews and approvals. Licenses and usage conditions will be documented separately for code, weights, metadata, and releasable samples.

We also plan to establish a public contact channel for content creators, rights holders, and affected individuals to request removal. Requests will be reviewed using source provenance records, and affected entries will be removed from hosted releases and reconstruction manifests where warranted. We will revise the Public designation in Table 1 and clarify the release conditions in the manuscript.

Q2

We evaluated OphthCLIP on 10 independently annotated ophthalmic classification datasets and MM-Retinal. The complete evaluation protocol, retrieval results, and caption refinement experiments are reported in our response to Reviewer bjw1 (W2/Q3).

We further separated the ten classification datasets into three OCT datasets (OCTID, OCTDL, Kermany) and seven fundus datasets (APTOS2019, IDRiD, MESSIDOR-2, PAPILA, GlaucomaFundus, Retina, JSIEC) to examine modality-specific transfer performance:

CategoryModelZS OCT (3)ZS Fundus (7)Probe OCT (3)Probe Fundus (7)
GeneralCLIP B/1622.228.178.460.3
CLIP L/1423.624.476.660.6
SigLIP L/1620.627.280.260.3
OphthalmologyEyeCLIP19.821.255.237.4
RET-CLIP‡15.523.382.970.6
FLAIR22.433.763.966.1
MedicalPMC-CLIP66.432.288.065.9
BiomedCLIP42.330.175.658.5
UniMed-CLIP†61.539.684.168.2
BMC-CLIP23.424.877.758.6
OursOphthCLIP XS (B/32)52.630.476.157.6
OphthCLIP S (B/16)47.827.282.764.5
OphthCLIP L (L/14)60.028.582.665.7
OphthCLIP H (H/14)54.830.686.267.8

All values are balanced accuracy percentages. ZS denotes zero-shot classification; Probe denotes linear probing. † UniMed-CLIP pretraining includes some evaluated data. ‡ RET-CLIP uses a Chinese text encoder and was evaluated with English prompts.

It is worth noting that OphthCLIP was trained on only 0.47M ophthalmic image–text pairs. The reported corpus sizes are over 5.3M for UniMed, 15M for PMC-15M (BiomedCLIP), and over 24M for the BIOMEDICA archive underlying BMC-CLIP.

References

OCT datasets. OCTID, OCTDL, Kermany OCT.

Fundus datasets. APTOS2019, IDRiD, MESSIDOR-2, PAPILA, Glaucoma Fundus, Retina (Cataract Dataset), JSIEC.

Baselines. CLIP, SigLIP, PMC-CLIP, BiomedCLIP, UniMed-CLIP, BMC-CLIP (BIOMEDICA), EyeCLIP, RET-CLIP, FLAIR.

Reviewer_bjw1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces Ophth-500K, a dataset of 467,697 ophthalmic image–text–audio samples curated from expert videos through automated filtering, alignment, and refinement. It validates the data by training OphthCLIP with standard image–text contrastive learning. On a 500-pair, video-disjoint validation set, the best variant achieves 32.6% and 33.2% Recall@1 for image-to-text and text-to-image retrieval, substantially outperforming the reported baselines. Validation is limited to internal image–text retrieval.

Critical strengths

1. A potentially valuable source of descriptive ophthalmic supervision at substantial scale. The dataset addresses a meaningful bottleneck: access to ophthalmic images paired with descriptions richer than disease labels. Extracting nearly 468,000 samples across three imaging modalities from expert videos could provide a useful resource for research. The integration of visual extraction, narration alignment, and description refinement constitutes a practical contribution, contingent on demonstrating data reliability and enabling meaningful access.

2. Strong initial evidence that the curated pairs contain learnable domain-specific signal. All four OphthCLIP variants outperform the reported baselines across both retrieval directions and all Recall@K metrics. The best model improves Recall@1 over the strongest baseline by 25.4 and 28.0 percentage points. Source-video separation makes this evidence more persuasive than a random frame-level split, although the demonstrated benefit remains specific to the reported evaluation setting.

Critical weaknesses

1. The central claim of reliable clinical supervision lacks quantitative validation. The pipeline combines several potentially error-prone stages, including transcription, fixed-window temporal alignment, region selection, and Qwen3-VL-based rewriting. The manuscript states that AI-assisted outputs were reviewed, and Figure 2 indicates expert verification, but it does not specify an auditable sampling protocol, reviewer qualifications, agreement, or measured rates of clinical error and image–description mismatch. Consequently, it is unclear whether the final descriptions faithfully preserve expert narration, introduce unsupported findings, or refer to content no longer present after cropping. Concept counts and illustrative examples do not resolve this uncertainty. Because the dataset is the primary contribution, this missing evidence directly limits confidence in its value.

2. The evaluation does not adequately establish generalization or isolate the benefits of the proposed curation. The only reported benchmark comprises 500 pairs drawn from the same dataset and processed through the same pipeline. There is no independently annotated external evaluation or ablation comparing refined descriptions with original transcripts. The paper also does not report matched-data adaptation controls that would help distinguish the benefits of the curation choices from domain-specific training more generally. Video-level separation is useful, but cross-video near-duplicate screening is not documented. Thus, the results demonstrate improved retrieval within this distribution, while leaving the broader representation-learning value and the contribution of individual processing stages unresolved.

Questions for authors
  1. What precisely does “expert verification” in Figure 2 involve? How many samples were examined, how were they selected, who reviewed them, and what proportions contained clinically incorrect descriptions, unsupported findings, or image–text mismatches? Please distinguish verification of the final samples from review of software or manuscript outputs.

  2. How many distinct videos and channels contribute to the 500 validation pairs, and how were these pairs selected? Was this set used for checkpoint or hyperparameter selection? Was cross-video image or slide duplication checked before splitting?

  3. Can the authors provide a controlled comparison of original transcripts versus refined descriptions, together with an evaluation on independently annotated external ophthalmic data? These would materially affect my assessment by testing whether the gains reflect improved clinical supervision and transferable representations. I view these as evidence needed to support the current contribution, rather than optional future extensions.

1 comment
AuthorsOpenReview ↗

Clinical supervision and evaluation

We thank the reviewer for raising these concerns.

W1/Q1:

Our keyframe extraction and audio alignment procedures largely follow Quilt-1M, including the reuse of several core implementations. Regarding clinical supervision and reliability, we audited randomly sampled 100 training and 100 evaluation samples. We also developed an annotation tool covering seven quality criteria:

Quality AssessmentTraining (n=100)Evaluation (n=100)
Ophthalmic image93% Valid, 7% Invalid100% Valid
Imaging modality27% Multimodal, 12% OCT, 26% CFP, 8% Angiography, 16% Others, 4% UWF31% Multimodal, 24% OCT, 15% CFP, 13% Angiography, 15% Others, 2% UWF
Crop quality76% Good, 15% Partial, 2% Poor86% Good, 14% Partial, 0% Poor
Image-text alignment83% Full, 6% Partial, 4% Mismatch87% Full, 12% Partial, 1% Mismatch
Invented content87% None, 5% Possible, 1% Confirmed93% None, 6% Possible, 1% Confirmed
Clinical accuracy89% Consistent, 4% No consensus91% Consistent, 9% No consensus
Patient-identifying information1%0%

Seven non-ophthalmic training samples were excluded by the annotation tool. All percentages are calculated over the original 100 sampled pairs.

Three independent reviewers (medical imaging researchers) assessed all 200 evaluation pairs. Inter-rater agreement was assessed using Gwet's AC1 and Fleiss' κ; disagreements were adjudicated by a senior clinician.

W2:

We thank the reviewer for this feedback. Regarding generalization, we evaluated the OphthCLIP on ten independently annotated ophthalmic classification datasets: OCTID, OCTDL, Kermany, APTOS2019, IDRiD, MESSIDOR-2, PAPILA, GlaucomaFundus, Retina, and JSIEC. We also evaluated cross-modal retrieval on 2,169 image-caption pairs from the CFP subset of MM-Retinal. All models used the same English prompts for zero-shot classification and logistic regression on L2-normalized image embeddings for linear probing, with regularization selected on validation data:

CategoryModelZero-shot bAccProbe bAccI2T R@10T2I R@10
GeneralCLIP B/1626.465.80.81.3
CLIP L/1424.265.40.90.9
SigLIP L/1625.266.31.42.1
OphthalmologyEyeCLIP20.842.70.50.4
RET-CLIP‡20.974.31.40.8
FLAIR†30.365.41.21.8
MedicalPMC-CLIP42.572.56.17.6
BiomedCLIP33.763.65.77.0
UniMed-CLIP†46.273.052.356.1
BMC-CLIP24.464.34.85.8
OursOphthCLIP XS (B/32)37.063.13.53.9
OphthCLIP S (B/16)33.469.95.25.8
OphthCLIP L (L/14)37.970.85.96.7
OphthCLIP H (H/14)37.973.35.96.9

Here, bAcc denotes mean balanced accuracy across the ten classification datasets. †^\dagger indicates overlap with some evaluation data during pretraining, including MM-Retinal for UniMed-CLIP. ‡^\ddagger RET-CLIP uses a Chinese text encoder, limiting interpretation under English prompts. We screened 8,747 external test images against the training set using feature cosine similarity. A total of 17 images had a training match with similarity ≥0.95\geq 0.95, and none reached 0.98.

Q2

The 500 validation pairs came from 459 videos across 247 YouTube channels. We filtered captions by 22 ophthalmic pathology terms, length (>8 words), and uniqueness, then selected the top 500 based on a heuristic score favoring specific, unambiguous descriptions.

We examined cross-video duplication using DINOv2 embeddings (cosine similarity ≥ 0.95). We identified 90 evaluation images had near-duplicates in the training set, primarily involving recurring lecture slides, reuploaded videos, or standard textbook figures. For instance, an identical fundus photograph from the MM-Retinal textbook collection appeared in one of our lecture videos.

We found no evidence of hyperparameter tuning using this validation set in the preserved training records. Each reported model has a single recorded training run, with configurations matching our experiment files (learning rate 5×10−65\times10^{-6}, weight decay 0.05, and 500 warmup steps), shared across backbones.

Q3:

Our early experiments using raw transcriptions as training captions showed limited cross-modal retrieval performance, partly due to noisy narration, irrelevant context, and inconsistent descriptions. This observation motivated us to introduce caption refinement into the curation pipeline. As shown below:

ModelTraining textI2T R@1I2T R@5I2T R@10T2I R@1T2I R@5T2I R@10
XSRaw transcription12.229.839.011.427.436.4
XSRefined description22.243.256.220.441.256.4
SRaw transcription17.835.647.615.635.845.4
SRefined description22.646.458.021.646.257.2
Reviewer_eFEwOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper builds a large ophthalmology image-text-audio dataset from 14,769 hours of expert lecture/demonstration videos using a pipeline that screens videos, segments visually stable clinical frames, aligns ASR narration, crops the relevant panel with SAM3 + BiomedCLIP gating, and rewrites the transcript into a short clinical description with a VLM.

Critical strengths

The resource itself is a real contribution — nothing public combines this scale with three imaging modalities, audio, and such broad language/geographic and UMLS concept coverage, and Table 1 makes that gap concrete. The engineering is also thoughtfully done (privacy masking, temporal window around each visual anchor, full-frame fallback), and it's reassuring that the backbone-size result is reported honestly rather than smoothed over.

Critical weaknesses

The evaluation is essentially self-serving: a 500-pair validation set drawn from the same curated corpus, so OphthCLIP is compared zero-shot-free against models that never saw this distribution — there's no downstream linear probe, zero-shot disease classification or retrieval on an external benchmark, which is what would actually justify the "foundation model" framing. There is also no human quality assessment of the pseudo-label. Finally, the licensing/provenance story for scraped video is thin, and "will be released" with no concrete license makes the dataset contribution hard to credit at review time.

Questions for authors

The evaluation is essentially self-serving: a 500-pair validation set drawn from the same curated corpus, so OphthCLIP is compared zero-shot-free against models that never saw this distribution — there's no downstream linear probe, zero-shot disease classification or retrieval on an external benchmark, which is what would actually justify the "foundation model" framing. There is also no human quality assessment of the pseudo-label. Finally, the licensing/provenance story for scraped video is thin, and "will be released" with no concrete license makes the dataset contribution hard to credit at review time.

1 comment
AuthorsOpenReview ↗

Justification of Foundation Model, Quality Assessment, and Dataset Availability

We thank the reviewer for the productive feedback.

External Evaluation and Generalization

To evaluate the representation learning capability of OphthCLIP beyond the original retrieval benchmark, we conducted additional zero-shot classification and linear-probe evaluations on ten independently annotated ophthalmic datasets, together with external retrieval on 2,169 image–caption pairs from MM-Retinal. The classification datasets were further grouped into three OCT and seven fundus datasets to examine transfer across imaging modalities.

ModelZS OCT (3)ZS Fundus (7)ZS All (10)Probe OCT (3)Probe Fundus (7)Probe All (10)MM-Retinal I2T R@10MM-Retinal T2I R@10
General
CLIP B/1622.228.126.478.460.365.80.81.3
CLIP L/1423.624.424.276.660.665.40.90.9
SigLIP L/1620.627.225.280.260.366.31.42.1
Ophthalmology
EyeCLIP19.821.220.855.237.442.70.50.4
RET-CLIP‡15.523.320.982.970.674.31.40.8
FLAIR†22.433.730.363.966.165.41.21.8
Medical
PMC-CLIP66.432.242.588.065.972.56.17.6
BiomedCLIP42.330.133.775.658.563.65.77.0
UniMed-CLIP†61.539.646.284.168.273.052.356.1
BMC-CLIP23.424.824.477.758.664.34.85.8
Ours
OphthCLIP-XS (B/32)52.630.437.076.157.663.13.53.9
OphthCLIP-S (B/16)47.827.233.482.764.569.95.25.8
OphthCLIP-L (L/14)60.028.537.982.665.770.85.96.7
OphthCLIP-H (H/14)54.830.637.986.267.873.35.96.9

All values are percentages. ZS and Probe denote zero-shot and linear-probe balanced accuracy. OCT includes OCTID, OCTDL, and Kermany; Fundus includes APTOS2019, IDRiD, MESSIDOR-2, PAPILA, Glaucoma_fundus, Retina, and JSIEC. MM-Retinal reports R@10 on 2,169 CFP image–caption pairs. † Pretraining includes some evaluation data (UniMed-CLIP includes MM-Retinal). ‡ RET-CLIP uses a Chinese text encoder, limiting interpretation under English prompts. Full results for individual datasets and the CFP, FFA, and OCT subsets of MM-Retinal are provided in the appendix.

These additional evaluations provide evidence of representation transfer across independent ophthalmic datasets using both zero-shot and supervised downstream evaluation. The evaluation protocol, duplication screening, and controlled caption refinement experiments are further documented in our response to Reviewer bjw1 (W2/Q3).

Human Quality Assessment

We conducted a systematic audit of randomly sampled 100 training pairs and 100 evaluation pairs across seven quality criteria. Three independent medical imaging researchers assessed the samples, with disagreements adjudicated by a senior clinician. The audit includes image validity, modality correctness, cropping, image–text alignment, invented content, clinical accuracy, and patient privacy, together with inter-rater agreement statistics. The complete audit protocol and results are reported in our response to Reviewer bjw1 (W1/Q1).

Dataset Availability and Licensing

Ophth-500K has not yet been publicly released. We plan a two-stage release strategy, subject to applicable source licenses, platform policies, institutional requirements, and relevant IRB requirements.

First, we plan to release the complete curation pipeline, source video identifiers, temporal annotations, processing configurations, and pretrained OphthCLIP weights through GitHub and Hugging Face, allowing reconstruction where source access and permissions permit.

Second, we plan to progressively release curated image–text pairs following individual manual review for clinical content, privacy, and redistribution eligibility, targeting approximately 50,000 samples per month subject to the required approvals.

We will separately document licensing conditions and provide a channel for content removal requests. Additional details regarding licensing, privacy, and the removal procedure are provided in our response to Reviewer fdQe (W2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.