acceptodds
Under review as a conference paper at ICLR 2027

Clinical Trajectory Alignment for Medical Vision-Language Pre-training

Abstract

Medical vision-language pre-training largely follows a visit-level image-report matching paradigm, aligning paired images and reports at individual visits. While effective for static cross-modal correspondence, this paradigm provides limited supervision for longitudinal clinical change, such as whether abnormalities improve, remain stable, or worsen over time. Learning such change is challenging because temporal semantics are implicit in free-text reports, and different abnormalities within the same patient may evolve asynchronously or even in opposite directions. We propose MedCTA, which reframes medical vision-language pre-training from visit-level cross-modal matching to learning clinical change. Rather than compressing a patient history into a single temporal representation, MedCTA models clinical change at two complementary scopes. At the abnormality scope, clinically grounded queries construct abnormality-conditioned visual and textual trajectories to capture heterogeneous abnormality evolution. At the patient-course scope, global image and report sequences are modeled to capture overall clinical progression beyond any individual abnormality. Structured trend supervision is extracted from longitudinal reports by an offline LLM parser, removing the need for manual temporal annotations. Combined with static image-report alignment, MedCTA learns representations that preserve visit-level cross-modal correspondence while encoding longitudinal change semantics. Experiments on temporal image classification, image-text retrieval, and zero-shot classification show consistent gains over strong medical vision-language baselines.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Medical Vision-Language Pre-training, Clinical Trajectory Modeling, Temporal Representation Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6dxUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

The proposed MedCTA framework supplements contrastive vision–language modelling in medical imaging with progression supervision from full patient trajectories. Global progression labels are extracted offline by an LLM from a sequence of reports, separately for specific abnormalities and for the full patient course. Abnormality-specific (computed via cross-attention) and patient-course representation trajectories are temporally aggregated then classified following the extracted labels. The method is evaluated on a suite of cross-modal and temporal tasks, and compared with multiple temporal and non-temporal VLM baselines. Reported results imply consistent improvements across all evaluated tasks.

Critical strengths

The work builds on a growing body of literature acknowledging the need to model temporality in radiology. Radiology scans are often reported incrementally, with explicit mentions of change, and sometimes cannot be interpreted without access to patient history (e.g. findings reported as "unchanged"). This is a crucial domain-specific consideration that is not addressed by conventional single-visit or general-domain methods.

The approach of explicitly modelling progression of individual abnormalities in addition to overall patient course is interesting and well motivated. Its proposed methodological implementation with learned queries and cross-attention over visual and textual tokens, followed by temporal classification, seems like an appropriate design. Even if generally there are important limitations to assuming a fixed panel of abnormalities, this signal is only used for additional supervision and is not necessarily detrimental for representation learning.

Critical weaknesses

One of my key concerns regards the suitability of the proposed temporal supervision scheme, which relies on trajectory-level progression labels. Trajectories are often non-monotonic, especially at the patient level (e.g. throughout a long ICU stay). This limitation is very briefly acknowledged in Appendix C.3 without further discussion. Why not supervise with visit-level progression labels, which seems perfectly compatible with the Mamba head? Relatedly, it is not clear what image representations are expected to gain from temporal supervision—can you guess from a single radiograph whether the patient is improving overall? Lastly, the authors provided virtually no detail about the labelling process itself (prompts, sampling settings, evaluation, handling of uncertain progression, etc.), making it impossible to assess or replicate.

My second major concern is that it is not clear what exactly is being demonstrated in the experiments. The authors have not specified what MedCTA representations are used in the downstream tasks: encoder global representations vs Mamba outputs? single- vs multi-visit? abnormality- vs patient-scope? etc. This is further disorienting because the inputs/outputs of the evaluation tasks are never fully stated. Moreover, the baselines were described at high level in Appendix A.3, but without mention of architectures, parameter counts, or training data, which can greatly confound the interpretation of results and may undermine methodological improvement claims. The dynamic phrase grounding evaluation comprises just three qualitative heatmap examples, with no attempt to quantify spatial localisation across a dataset or compare with other models. Finally, discussion of results for each experiment was generally quite shallow, with limited insight on how the method's components might have led to improved results on each task.

Questions for authors
  1. The assumption of monotonic progression at both abnormality- and patient-scope needs further justification, and its trade-offs should be discussed.
  2. Because the LLM parsing is listed as a core contribution, the process must be described in detail to allow judging its appropriateness. Also, please clarify whether the labelling itself was evaluated for correctness against human ground truth (not just downstream performance).
  3. Specify precisely how the MedCTA model interfaces with each of the downstream tasks (inputs/outputs).
  4. Add the missing details for the chosen baselines, discuss possible confounders, and contextualise the fairness of the reported comparisons. If the baselines were retrained by the authors with similar settings to MedCTA, this must be stated explicitly, as it would significantly strengthen the reported evidence.
  5. Elaborate on the discussion of experimental findings: which design choices improved results and why (also contextualising with the ablations)? how does it compare to the baselines, and are there any patterns/trends amongst them? what are limitations of the proposed method? etc.
9 comments

Response to W1, Q5:

(1) Non-monotonic Temporal Supervision

MedCTA learns the dominant direction of change over the entire patient observation window, allowing disease progression to exhibit non-monotonic changes across different visits. Each pre-training sequence contains up to four consecutive visits. Trend annotations are determined based on descriptions explicitly supported by clinical reports, following the priority of magnitude of change, persistence of change, and overall changes in major abnormalities to determine the overall trend within the observation window. We have added the prompt template for extracting temporal trend labels to the revised manuscript. During pre-training, image and text features from multiple visits are extracted through shared encoders and are used to construct patient-level and abnormality-level trajectory sequences, respectively, which are then temporally modeled using Mamba. Subsequently, patient-level and disease-level trend labels are used to supervise the corresponding trajectory representations. Through this temporal supervision, MedCTA learns longitudinal information about both overall patient disease progression and disease-specific changes.

To further evaluate whether the representations learned through the aforementioned trend supervision can transfer to multi-disease longitudinal image understanding tasks, we conduct an exploratory evaluation on the Global Trajectory Summarization multi-abnormality subset of MI-CXR [1]. Each sample contains five temporally ordered chest X-rays, one question, and five candidate longitudinal summaries. The task is to select the summary that best matches the evolution of multiple abnormalities. The data were split into training, validation, and test sets at the patient level, with proportions of 80%, 10%, and 10%, respectively. The image and text encoders were frozen to extract features for the five images and five candidate answers. The image features were average-pooled and mapped through a trainable projection head, followed by cosine similarity computation with the candidate text features. During training, the index of the correct answer was used as supervision, and only the projection head was optimized using a five-candidate cross-entropy loss. During testing, answers were selected and ranked according to their similarity scores.

As shown in Table 1, MedCTA achieves an ACC of 63.30% and an MRR of 79.03%, improving over TAMM by 1.50 and 1.92 percentage points, respectively. F1 and AUROC are also higher. The improvements in ACC and MRR jointly demonstrate advantages in both correct summary selection and ranking, providing exploratory evidence of the model's ability to represent longitudinal disease evolution. We have added the global trajectory summary matching task to Section 4.2.1 of the revised manuscript.

Table 1. Candidate-answer matching on the MI-CXR GTS multi-abnormality set (%).

MethodACCF1AUROCMRR
TAMM61.8059.3686.9577.11
MedCTA63.3062.0388.8879.03

[1] MI-CXR: A Benchmark for Longitudinal Reasoning over Multi-Interval Chest X-rays. ACL 2026.

(2) LLM Annotation Details

The offline Qwen2-7B-Instruct model processes complete reports arranged in chronological order and generates trend labels for the patient level and twelve abnormality categories: improvement 1, stability 0, and worsening -1. The abnormality loss is computed only for valid labels, while non-monotonic processes are summarized according to the aforementioned dominant-trend criteria.

To further assess the clinical correctness of the trend labels generated by the LLM, we conducted a manual annotation-based evaluation. We randomly sampled 100 longitudinal patient sequences, which were independently annotated by two authors. The average agreement rate between LLM-generated and human-annotated patient-level trend labels was 76%, while the inter-annotator agreement rate was 72%. For disease-level trend labels, the corresponding agreement rates were 69% and 73%, respectively. The agreement between the LLM and human annotations is comparable to the inter-annotator agreement, providing preliminary quantitative support for label quality.

Compared with patient-level trend labeling, disease-level trend recognition requires not only distinguishing among different disease types but also independently determining whether each disease is improving, stable, or worsening. This involves finer-grained semantic understanding and multidimensional judgment, and therefore entails greater annotation complexity. We have added the LLM annotation details to Appendix C.4 of the revised manuscript.

Response to W1, Q5:

(3) Trajectory-level and Visit-level Progression Labels

In clinical practice, understanding both disease-specific evolution and overall patient disease progression is important for longitudinal clinical assessment. Visit-level supervision, which focuses on a single visit or changes between adjacent visits, primarily captures local changes and cannot directly constrain the overall direction of evolution across the entire patient observation window.

To address this issue, MedCTA proposes an LLM-driven joint trajectory learning framework. Through disease-level and patient-level trend supervision, it jointly learns disease-specific evolution representations and overall patient progression representations, thereby extending local visit-level change modeling to comprehensive clinical trajectory learning across the entire observation window.

To evaluate the contributions of complete sequence information and explicit temporal modeling to pre-trained representations, we introduce two comparison variants: First–Last and Mean Pooling. While keeping the training data, both types of trend labels, and supervision losses consistent, First–Last feeds only the first and last visits into the temporal branch, with both image and report sequence lengths set to 2. Mean Pooling retains the complete sequence, replaces the temporal encoder with average pooling along the temporal dimension, and performs the same trend prediction.

As shown in Tables 2 and 3, the complete MedCTA consistently outperforms both variants across all metrics. Compared with First–Last, MedCTA improves zero-shot ACC by 3.87 and 3.25 percentage points on COVIDx and RSNA, respectively, and improves linear-probe ACC by 3.26 and 0.62 percentage points, respectively. These results indicate that intermediate visit information contributes to learning more comprehensive longitudinal clinical representations. Furthermore, the complete MedCTA outperforms Mean Pooling, further demonstrating the contribution of explicit temporal modeling over simple temporal average aggregation. We have added the First–Last and Mean Pooling baselines to Section 4.3 of the revised manuscript.

Table 2. Zero-shot classification of pre-training variants on COVIDx and RSNA Pneumonia (%).

MethodACC (COVID)F1 (COVID)AUROC (COVID)ACC (RSNA)F1 (RSNA)AUROC (RSNA)
first_last74.1560.4586.8564.9755.1185.60
mean_pooling76.7662.8487.4266.5156.3787.49
MedCTA78.0263.9389.6268.2257.9188.60

Table 3. Linear-probe classification of pre-training variants on COVIDx and RSNA Pneumonia (%).

MethodACC (COVID)F1 (COVID)AUROC (COVID)ACC (RSNA)F1 (RSNA)AUROC (RSNA)
first_last92.4083.3796.6385.4377.6989.65
mean_pooling93.5484.6195.5285.2977.5389.57
MedCTA95.6688.4896.9286.0579.2589.82

Response to W2, Q3, Q4, Q5:

(1) Details of Downstream Tasks

  • Temporal Image Classification: The pre-trained image encoder is frozen, and only the classification head is trained on Chest ImaGenome. The current and previous chest X-rays are processed by the shared encoder to generate image-level features, which are concatenated in the order of "current, previous" to predict the corresponding abnormality trend, including improvement, stability, or worsening. During MS-CXR-T testing, both the image encoder and classification head are frozen. This task uses image-level features.
  • Temporal Sentence Similarity Classification: Two text sentences are separately encoded by the frozen text encoder to extract global features. After L2 normalization, cosine similarity is calculated, and ACC and AUROC are used for evaluation.
  • Phrase Grounding: The image and text encoders are frozen. Similarities between image patch features and text features are used to generate heatmaps, and IoU, Dice, and CNR are reported in conjunction with manually annotated bounding boxes. As shown in Table 4, MedCTA achieves higher values than TAMM on all three metrics, with the main improvement observed in CNR.

Table 4. Quantitative phrase-grounding results.

MethodIoUDiceCNR
TAMM0.11080.18940.4043
MedCTA0.11240.19170.5895
  • Image Classification: In the original RSNA/COVIDx experiments, the data were split into training, validation, and test sets at a ratio of 80%/10%/10%. The image encoder was frozen, and the linear classifier was fine-tuned. Therefore, this task is Linear Probe. In the newly added zero-shot experiments, both the image and text encoders are frozen, and classification is performed based on the similarity between images and class texts, without training a classifier using target labels. As shown in Table 5, MedCTA outperforms TAMM and ALTA on ACC, F1, and AUROC across both datasets.

Table 5. Zero-shot image–text classification on COVIDx and RSNA Pneumonia (%).

MethodACC (COVID)F1 (COVID)AUROC (COVID)ACC (RSNA)F1 (RSNA)AUROC (RSNA)
ALTA62.7049.0385.9565.4056.6982.13
TAMM76.1259.1986.1566.8556.1981.62
MedCTA78.0263.9389.6268.2257.9188.60

(2) Experimental Data Splits

When constructing the MedCTA temporal pre-training dataset, we excluded downstream patients based on patient IDs (subject_id). MS-CXR-T, MIMIC-5×200, and the Chest ImaGenome subset used in our experiments involve 800, 957, and 16,001 patients, respectively. We took the union of the patient sets from these three datasets and excluded all images and reports belonging to these patients from MIMIC-CXR before constructing longitudinal sequences. The resulting pre-training dataset contains 46,564 patients and 61,164 sequences. We have added details of the data split to Appendix A.1 of the revised manuscript.

(3) Baseline Details

We categorize the baseline results according to their sources:

  • Quoted results: ACC for temporal image classification, ACC for temporal sentence similarity classification, cross-modal retrieval metrics, and ACC and F1 for RSNA/COVIDx linear probing are quoted from the TAMM paper.
  • Reproduced results: AUROC for temporal sentence similarity classification and RSNA/COVIDx classification. Specifically, TAMM was retrained following its original method on the same patient-excluded data as MedCTA, while the remaining baselines were evaluated using their official pre-trained weights.

MedCTA follows the initialization strategy of TAMM, using publicly available MGCA weights for initialization, followed by temporal pre-training on the patient-excluded dataset. The initialization of the remaining baselines follows the experimental configurations reported in the TAMM paper. The parameter counts of all models are reported in Table 6. We have added the baseline details to Appendix A.3 of the revised manuscript.

Table 6. Model parameter counts (M)

MethodParameters (M)
MGCA172.73
MedCLIP133.26
PRIOR159.92
MAVL170.65
CARZero220.77
Med-ST233.76
ALTA265.65
TAMM321.91
MedCTA198.70
Reviewer_WBuEOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MedCTA, a medical vision-language pre-training framework for learning longitudinal clinical change from chest X-ray and report sequences. It models abnormality-specific evolution through abnormality-conditioned trajectories constructed with learnable queries, and patient-course progression through sequences of global image and report features. Both are encoded with Mamba and supervised using LLM-derived trend labels: improving, stable, or worsening. These objectives are combined with static image–report alignment.

Critical strengths
  • Modeling each abnormality separately is well motivated. Different abnormalities in the same patient can progress in different directions. Constructing a separate trajectory for each abnormality is a reasonable way to preserve these differences, which may be lost in a single patient-level representation.
Critical weaknesses
  • The experiments do not directly establish the main contribution beyond TAMM [1]. TAMM already uses LLM-derived three-class trend labels to supervise trend prediction in both modalities. MedCTA extends this approach with abnormality-conditioned trajectories and patient-course progression modeling, but the experiments do not directly evaluate patient-course trend prediction or cases where abnormalities progress in different directions. The loss ablations, query initialization comparison, and visit-order perturbation provide useful evidence, but do not isolate the benefit of abnormality queries themselves or of using Mamba rather than a simpler temporal encoder.
  • It is unclear whether the baseline results were obtained under a common evaluation protocol. Section 4.1 states that baselines are reproduced using official implementations where available, yet all baseline results in Table 1, including standard deviations, exactly match those reported in TAMM. This needs clarification: were these results reproduced or taken from prior work? Given the modest improvements over TAMM on temporal image classification, uncertainty about the evaluation protocol makes these gains difficult to assess.

[1] Bai, Liang, et al. “Medical Vision–Language Pretraining with LLM-Guided Temporal Supervision.” Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2026.

Questions for authors
  1. Which baseline results were reproduced under the current evaluation protocol, and which were taken from prior work? Please also explain the discrepancy between the MGCA Consolidation result in Table 1 (44.99 ± 0.47) and that reported in Med-ST (50.79 ± 0.44).
  2. MS-CXR-T, Chest ImaGenome, and MIMIC-5×200 are derived from MIMIC-CXR. Which patients or studies were excluded from pre-training to prevent overlap with downstream evaluation data?
  3. Table 3 is described as zero-shot classification, but Appendix A.4 states that a lightweight classifier is trained on the pre-trained representations. What data and labels are used to train this classifier, and in what sense is the evaluation zero-shot?
  4. Appendix C.3 mentions a manual evaluation. Could you report agreement between the LLM-generated labels and human annotations for both abnormality-specific and patient-course trends? Please also specify the abnormality set and the label distributions for both types of supervision.
  5. How does performance change when abnormality queries are removed or Mamba is replaced with a simpler temporal encoder?
  6. How is the pre-trained sequence model adapted to the two-image setting of MS-CXR-T? Please clarify how the image pair is encoded and which components are fine-tuned.
9 comments

Response to W1, Q5:

(1) Core Contributions of MedCTA and Supporting Experimental Evidence

Thank you for your suggestions. We would like to clarify that TAMM is a relatively simple and coarse-grained approach that uses an LLM to identify trend changes between two adjacent visits. However, it does not construct disease trajectories and therefore cannot capture the dynamic process of disease evolution over time. In contrast, MedCTA constructs both disease-level and patient-level trajectories across multiple visits and uses an LLM to identify trend changes across the entire multi-visit sequence as supervision signals for model training. This enables the model to capture disease evolution and makes MedCTA more suitable for trend modeling in medical vision-language pre-training.

To explore MedCTA’s ability to represent longitudinal disease evolution, we conducted candidate-summary matching on the Global Trajectory Summarization multi-abnormality subset of MI-CXR [1]. Each sample contains five temporally ordered chest X-rays, a question, and five candidate summaries describing the evolution of multiple abnormalities. Data were split by patient into training, validation, and test sets (80%/10%/10%). With both image and text encoders frozen, only a projection head was trained to map features averaged across the five images into the text feature space. Training used five-class cross-entropy on image–summary cosine similarities, with the correct summary index as the target; testing ranked candidates by similarity and selected the best match. As shown in Table 1, MedCTA achieves ACC, F1, AUROC, and MRR of 63.30%, 62.03%, 88.88%, and 79.03%, respectively, outperforming TAMM (61.80%, 59.36%, 86.95%, and 77.11%). The respective ACC and MRR gains of 1.50 and 1.92 percentage points indicate improved correct summary identification and candidate ranking, providing exploratory evidence of MedCTA’s ability to represent longitudinal disease evolution. We have added this task to Section 4.2.1 of the revised manuscript.

Table 1. Candidate-answer matching on the MI-CXR GTS multi-abnormality set (%).

MethodACCF1AUROCMRR
TAMM61.8059.3686.9577.11
MedCTA63.3062.0388.8879.03

[1] MI-CXR: A Benchmark for Longitudinal Reasoning over Multi-Interval Chest X-rays. ACL 2026.

(2) Additional Ablation Study on Abnormality Queries

We added a variant without abnormality queries, which directly uses the image patch and text token representations from each visit to construct visual and textual sequences in chronological order, followed by temporal encoding.

As shown in Tables 2 and 3, MedCTA outperforms the variant without abnormality queries across all metrics on both datasets in zero-shot and linear-probing classification. These results highlight the benefits of abnormality-conditioned feature extraction for classification performance and representation quality. We have added this ablation study to Section 4.3 of the revised manuscript.

Table 2. Zero-shot classification with and without abnormality queries (%)

MethodACC (COVID)F1 (COVID)AUROC (COVID)ACC (RSNA)F1 (RSNA)AUROC (RSNA)
w/o Abnormality Queries76.9363.0189.0667.2757.0287.89
MedCTA78.0263.9389.6268.2257.9188.60

Table 3. Linear probing with and without abnormality queries (%)

MethodACC (COVID)F1 (COVID)AUROC (COVID)ACC (RSNA)F1 (RSNA)AUROC (RSNA)
w/o Abnormality Queries94.7787.2696.8585.8578.8189.78
MedCTA95.6688.4896.9286.0579.2589.82

(3) Additional Ablation Study on Temporal Encoders

We replaced Mamba with GRU while keeping abnormality queries, supervision labels, training objectives, and all other experimental settings unchanged. Mamba improves the eight retrieval metrics by 0.51–2.75 percentage points (Table 4). For temporal sentence similarity classification, ACC and AUROC improve by 4.75 and 12.96 percentage points, respectively (Table 5). These results support that, under the current configuration, Mamba learns representations that are more beneficial for retrieval and temporal semantic matching. We have added details of the ablation study on temporal encoders to Section 4.3 of the revised manuscript.

Table 4. Cross-modal retrieval with Mamba and GRU (%)

Retrieval MetricGRUMedCTA
T→I P@178.3180.24
T→I P@278.4679.08
T→I P@573.2175.68
T→I P@1063.0765.82
I→T P@167.7569.13
I→T P@268.1268.63
I→T P@566.2067.23
I→T P@1061.8863.01

Table 5. Temporal sentence similarity classification with Mamba and GRU (%)

MethodACCAUROC
GRU90.9785.16
MedCTA95.7298.12

Response to W2, Q1, Q2, Q3:

(1) Baseline Details

We categorize the baseline results according to their sources:

  • Quoted results: ACC for temporal image classification, ACC for temporal sentence similarity classification, cross-modal retrieval metrics, and ACC and F1 for RSNA/COVIDx linear probing are quoted from the TAMM paper.
  • Reproduced results: AUROC for temporal sentence similarity classification and RSNA/COVIDx classification. Specifically, TAMM was retrained following its original method on the same patient-excluded data as MedCTA, while the remaining baselines were evaluated using their official pre-trained weights.

MedCTA follows the initialization strategy of TAMM, using publicly available MGCA weights for initialization, followed by temporal pre-training on the patient-excluded dataset. The initialization of the remaining baselines follows the experimental configurations reported in the TAMM paper. The parameter counts of all models are reported in Table 6. We have added the baseline details to Appendix A.3 of the revised manuscript.

Table 6. Model parameter counts (M)

MethodParameters (M)
MGCA172.73
MedCLIP133.26
PRIOR159.92
MAVL170.65
CARZero220.77
Med-ST233.76
ALTA265.65
TAMM321.91
MedCTA198.70

(2) Experimental Data Splits

When constructing the MedCTA temporal pre-training dataset, we excluded downstream patients based on patient IDs (subject_id). MS-CXR-T, MIMIC-5×200, and the Chest ImaGenome used in our experiments involve 800, 957, and 16,001 patients, respectively. We took the union of the patient sets from these three datasets and excluded all images and reports belonging to these patients from MIMIC-CXR before constructing longitudinal sequences. The resulting pre-training dataset contains 46,564 patients and 61,164 sequences. We have added details of the data split to Appendix A.1 of the revised manuscript.

(3) Zero-shot Evaluation Protocol

In the original manuscript, the RSNA Pneumonia and COVIDx experiments used training labels from the target datasets and trained only a linear classifier while keeping the image encoder frozen. These experiments should therefore be referred to as Linear Probe. Similarly, works such as TAMM (AAAI 2026) also adopt Linear Probe to evaluate the representation capability of pre-trained image encoders on downstream tasks. We have consistently revised the main text, table captions, and related conclusions to distinguish Linear Probe from Zero-shot Evaluation.

We have also added a fully zero-shot evaluation: frozen pre-trained image and text encoders are used to extract image and class-text features, respectively, followed by classification based on cosine similarity. All baseline models adopt the same evaluation protocol. As shown in Table 7, MedCTA outperforms TAMM and ALTA on ACC, F1, and AUROC across both datasets. We have revised the content of Section 4.2.2 and the relevant table captions in the revised manuscript.

Table 7. Zero-shot image classification on COVIDx and RSNA (%)

MethodACC (COVID)F1 (COVID)AUROC (COVID)ACC (RSNA)F1 (RSNA)AUROC (RSNA)
ALTA62.7049.0385.9565.4056.6982.13
TAMM76.1259.1986.1566.8556.1981.62
MedCTA78.0263.9389.6268.2257.9188.60

Response to Q4, Q6:

(1) Evaluation of the Correctness of LLM-generated Labels

To further assess the clinical correctness of the trend labels generated by the LLM, we conducted a manual annotation-based evaluation. We randomly sampled 100 longitudinal patient sequences, which were independently annotated by two authors. The average agreement rate between LLM-generated and human-annotated patient-level trend labels was 76%, while the inter-annotator agreement rate was 72%. For disease-level trend labels, the corresponding agreement rates were 69% and 73%, respectively. The agreement between the LLM and human annotations is comparable to the inter-annotator agreement, providing preliminary quantitative support for label quality.

Compared with patient-level trend labeling, disease-level trend recognition requires not only distinguishing among different disease types but also independently determining whether each disease is improving, stable, or worsening. This involves finer-grained semantic understanding and multidimensional judgment, and therefore entails greater annotation complexity. We have added the evaluation of the correctness of LLM-generated labels to Appendix C.4 of the revised manuscript.

(2) Abnormality Category Set

The pre-training data cover 12 abnormality categories, including: Enlarged Cardiomediastinum, Cardiomegaly, Lung Lesion, Lung Opacity, Edema, Consolidation, Pneumonia, Atelectasis, Pneumothorax, Pleural Effusion, Pleural Other, and Fracture.

To further characterize the composition of longitudinal trend labels in the pre-training data, we counted the distributions of patient-level and individual disease-level trend labels across different directions of change, as shown in Table 8. Here, 1, 0, and -1 represent improvement, stability, and worsening, respectively. We have added details of the abnormality category set to Appendix C.5 of the revised manuscript.

Table 8. Patient-level and abnormality-level trend label distributions

Scope / AbnormalityImprove (1)Stable (0)Worse (-1)
Patient204781554725139
Cardiomegaly390253032928
Enlarged Cardiomediastinum1088243195214
Lung Lesion11539932136
Lung Opacity478645648316
Edema275642634321
Consolidation432442674972
Pneumonia345634744276
Atelectasis951266567697
Pneumothorax9959009930
Pleural Effusion7022119136772
Pleural Other204224503748
Fracture10023659929

(3) Dual-image Adaptation for MS-CXR-T

We first use the Chest ImaGenome dataset to construct training and validation sets, freeze the pre-trained image encoder, and fine-tune only the classification head. During testing on MS-CXR-T, both the image encoder and the classification head fine-tuned on Chest ImaGenome are frozen, with no further parameter updates.

For each sample, the current chest X-ray and the previous chest X-ray are separately fed into the image encoder with shared weights to extract image-level features. These features are then concatenated along the feature dimension in a fixed order of "current image, previous image" to construct a paired-image representation. Subsequently, the paired-image representation is fed into the classification head to predict the corresponding abnormality trend category, including improvement, stability, and worsening. We have added the corresponding content to Section 4.2.1 of the revised manuscript.

Reviewer_Mw8JOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MedCTA, a medical vision-language pre-training framework that combines static image–report alignment with abnormality-specific and patient-level temporal modeling, supervised by LLM-generated trend labels. Experiments on temporal classification, cross-modal retrieval, and disease classification generally outperform the reported baselines, while ablations support the contribution of both temporal objectives.

Critical strengths

1. The dual-scope formulation addresses a meaningful limitation of longitudinal representation learning. Modeling each abnormality separately while retaining a patient-level progression branch is well motivated by heterogeneous clinical evolution. This is more informative than assigning a single undifferentiated representation to the entire patient history. Removing either temporal objective degrades performance in the reported temporal-task ablations, providing evidence that both supervision scopes are useful within the proposed framework.

2. The empirical evaluation covers both temporal understanding and conventional cross-modal tasks. MedCTA improves all five reported temporal image classification categories and seven of eight retrieval metrics, while showing a substantial improvement in temporal sentence similarity accuracy over TAMM (95.72% versus 88.88%). The parser-replacement, order-perturbation, and label-noise experiments provide additional checks beyond the main comparisons. Together, these results suggest that the proposed supervision can improve representations across multiple tasks, although the image-classification gains are more modest than the language-side gains.

Critical weaknesses

1. The evaluation protocol is insufficiently specified to establish the interpretation and fairness of several key results. Section 4.2.2 describes RSNA Pneumonia and COVIDx evaluation as zero-shot, whereas Appendix A.4 states that these experiments involve training a lightweight classifier. Whether this classifier uses target-dataset labels fundamentally changes the claim being evaluated. In addition, Appendix A.2 states that downstream data are disjoint from pre-training data, but does not specify patient-level exclusions or how overlap is prevented across the MIMIC-derived benchmarks. Since MedCTA also starts from MGCA weights, the initialization and subsequent training budgets used for the baselines need to be explicit. These omissions prevent a confident assessment of the reported transfer gains.

2. The evidence does not sufficiently isolate the claimed benefit of trajectory modeling from that of additional coarse trend supervision. Each abnormality trajectory and patient course is ultimately supervised by a single three-class label. The paper does not compare its temporal encoders against simpler aggregation or first–last-visit alternatives under otherwise identical supervision. Shuffling visits while retaining the original labels disrupts the relationship between inputs and targets; the resulting degradation establishes sensitivity to this disruption but does not by itself demonstrate the necessity of the proposed temporal architecture. Furthermore, reversal consistency and synthetic random-noise tests do not establish the clinical correctness of the original LLM labels. The acknowledged non-monotonic failure case illustrates why this matters. These limitations weaken the stronger interpretation that the framework learns detailed clinical trajectories, rather than useful sequence-level trend classification.

Questions for authors
  1. For RSNA Pneumonia and COVIDx, what exactly is trained, on which data and labels, and which parameters are updated? How should the statement in Appendix A.4 about training a lightweight classifier be reconciled with the zero-shot designation? Were all baselines evaluated using precisely the same protocol?

  2. Please specify the pre-training patient/sequence counts and patient-level exclusion procedure for the downstream benchmarks. Does the non-overlap guarantee also cover the data used to train the MGCA initialization? What initialization, training data, and additional optimization budget were used for each baseline?

  3. Was a representative subset of abnormality-level and patient-level labels independently assessed by human annotators? If so, please report the sample size, annotation criteria, and agreement with the LLM. Please also provide the extraction prompts and explain how mixed or non-monotonic trajectories are assigned a single label, and how pre-existing temporal comparison language is handled during reversal filtering.

2 comments

Response to W1, Q1, Q2:

(1) Zero-shot Evaluation Protocol

In the original manuscript, the RSNA Pneumonia and COVIDx experiments used training labels from the target datasets and trained only a linear classifier while keeping the image encoder frozen. These experiments should therefore be referred to as Linear Probe. Similarly, works such as TAMM (AAAI 2026) also adopt Linear Probe to evaluate the representation capability of pre-trained image encoders on downstream tasks. We have also added a fully zero-shot evaluation: frozen pre-trained image and text encoders are used to extract image and class-text features, respectively, followed by classification based on cosine similarity. All baseline models adopt the same evaluation protocol. As shown in Table 1, MedCTA outperforms TAMM and ALTA on ACC, F1, and AUROC across both datasets. We have revised the content of Section 4.2.2 and the relevant table captions in the revised manuscript.

Table 1. Zero-shot image classification on COVIDx and RSNA (%)

MethodACC (COVID)F1 (COVID)AUROC (COVID)ACC (RSNA)F1 (RSNA)AUROC (RSNA)
ALTA62.7049.0385.9565.4056.6982.13
TAMM76.1259.1986.1566.8556.1981.62
MedCTA78.0263.9389.6268.2257.9188.60

(2) Experimental Data Splits

When constructing the MedCTA temporal pre-training dataset, we excluded downstream patients based on patient IDs (subject_id). MS-CXR-T, MIMIC-5×200, and the Chest ImaGenome used in our experiments involve 800, 957, and 16,001 patients, respectively. We took the union of the patient sets from these three datasets and excluded all images and reports belonging to these patients from MIMIC-CXR before constructing longitudinal sequences. The resulting pre-training dataset contains 46,564 patients and 61,164 sequences. We have added details of the data split to Appendix A.1 of the revised manuscript.

(3) Baseline Details

We categorize the baseline results according to their sources:

  • Quoted results: ACC for temporal image classification, ACC for temporal sentence similarity classification, cross-modal retrieval metrics, and ACC and F1 for RSNA/COVIDx linear probing are taken from the TAMM paper.
  • Reproduced results: AUROC for temporal sentence similarity classification and RSNA/COVIDx classification. Specifically, TAMM was retrained following its original method on the same patient-excluded data as MedCTA, while the remaining baselines were evaluated using their official pre-trained weights.

MedCTA follows the initialization strategy of TAMM, using publicly available MGCA weights for initialization, followed by temporal pre-training on the patient-excluded dataset. The initialization of the remaining baselines follows the experimental configurations reported in the TAMM paper. The parameter counts of all models are reported in Table 2. We have added the baseline details to Appendix A.3 of the revised manuscript.

Table 2. Model parameter counts (M)

MethodParameters (M)
MGCA172.73
MedCLIP133.26
PRIOR159.92
MAVL170.65
CARZero220.77
Med-ST233.76
ALTA265.65
TAMM321.91
MedCTA198.70

Response to W2, Q3:

(1) Contribution of Temporal Modeling

We added First–Last and Mean Pooling baselines while keeping the training data, patient-level and disease-level three-class labels, and losses consistent. First–Last feeds only the first and last visits into the temporal branch, with both image and report sequence lengths set to 2. Mean Pooling retains the complete sequence, replaces the temporal encoder with average pooling along the temporal dimension, and performs the same trend prediction.

As shown in Tables 3 and 4, the complete MedCTA consistently outperforms both variants across all metrics. Compared with First–Last, MedCTA improves zero-shot ACC by 3.87 and 3.25 percentage points on COVIDx and RSNA, respectively, and improves linear-probe ACC by 3.26 and 0.62 percentage points, respectively. These results indicate that intermediate visit information contributes to learning more comprehensive longitudinal clinical representations. Furthermore, the complete MedCTA outperforms Mean Pooling, further demonstrating the contribution of explicit temporal modeling over simple temporal average aggregation. We have added details of the First–Last and Mean Pooling baselines to Section 4.3 of the revised manuscript.

Table 3. Zero-shot classification of pre-training variants on COVIDx and RSNA (%)

MethodACC (COVID)F1 (COVID)AUROC (COVID)ACC (RSNA)F1 (RSNA)AUROC (RSNA)
first_last74.1560.4586.8564.9755.1185.60
mean_pooling76.7662.8487.4266.5156.3787.49
MedCTA78.0263.9389.6268.2257.9188.60

Table 4. Linear probing of pre-training variants on COVIDx and RSNA (%)

MethodACC (COVID)F1 (COVID)AUROC (COVID)ACC (RSNA)F1 (RSNA)AUROC (RSNA)
first_last92.4083.3796.6385.4377.6989.65
mean_pooling93.5484.6195.5285.2977.5389.57
MedCTA95.6688.4896.9286.0579.2589.82

(2) Clinical Correctness of LLM-generated Labels

To further assess the clinical correctness of the trend labels generated by the LLM, we conducted a manual annotation-based evaluation. We randomly sampled 100 longitudinal patient sequences, which were independently annotated by two authors. The average agreement rate between LLM-generated and human-annotated patient-level trend labels was 76%, while the inter-annotator agreement rate was 72%. For disease-level trend labels, the corresponding agreement rates were 69% and 73%, respectively. The agreement between the LLM and human annotations is comparable to the inter-annotator agreement, providing preliminary quantitative support for label quality. We have added details of the Clinical Correctness of LLM-generated Labels to Appendix C.4 of the revised manuscript.

(3) Evaluation of Representation Capability for Disease Evolution and Clinical Trajectories

We conducted an exploratory candidate-answer matching evaluation on the Global Trajectory Summarization (GTS) multi-abnormality subset of MI-CXR [1]. Each sample contains five temporally ordered chest X-rays, one question, and five candidate longitudinal summaries, covering the evolution of multiple abnormalities at different time points. The task requires selecting the summary that best matches the image sequence.

The data were split into training, validation, and test sets at the patient level, with proportions of 80%, 10%, and 10%, respectively. The image and text encoders were frozen to extract features for the five images and five candidate answers. The image features were average-pooled and mapped through a trainable projection head, followed by cosine similarity computation with the candidate text features. During training, the index of the correct answer was used as supervision, and only the projection head was optimized using a five-candidate cross-entropy loss. During testing, answers were selected and ranked according to their similarity scores.

As shown in Table 5, MedCTA achieves an ACC of 63.30% and an MRR of 79.03%, improving over TAMM by 1.50 and 1.92 percentage points, respectively. F1 and AUROC are also higher. The improvements in ACC and MRR jointly demonstrate advantages in both correct summary selection and ranking, providing exploratory evidence of the model's ability to represent longitudinal disease evolution. We have added the global trajectory summary matching task to Section 4.2.1 of the revised manuscript.

Table 5. Candidate-answer matching on the MI-CXR GTS multi-abnormality set (%)

MethodACCF1AUROCMRR
TAMM61.8059.3686.9577.11
MedCTA63.3062.0388.8879.03

[1] MI-CXR: A Benchmark for Longitudinal Reasoning over Multi-Interval Chest X-rays. ACL 2026.

(4) Prompt Template

We have added details of Prompt Template to Appendix E of the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.