- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3(+1)
- Soundness
- 2
Read review
Paper summary
The proposed MedCTA framework supplements contrastive vision–language modelling in medical imaging with progression supervision from full patient trajectories. Global progression labels are extracted offline by an LLM from a sequence of reports, separately for specific abnormalities and for the full patient course. Abnormality-specific (computed via cross-attention) and patient-course representation trajectories are temporally aggregated then classified following the extracted labels. The method is evaluated on a suite of cross-modal and temporal tasks, and compared with multiple temporal and non-temporal VLM baselines. Reported results imply consistent improvements across all evaluated tasks.
Critical strengths
The work builds on a growing body of literature acknowledging the need to model temporality in radiology. Radiology scans are often reported incrementally, with explicit mentions of change, and sometimes cannot be interpreted without access to patient history (e.g. findings reported as "unchanged"). This is a crucial domain-specific consideration that is not addressed by conventional single-visit or general-domain methods.
The approach of explicitly modelling progression of individual abnormalities in addition to overall patient course is interesting and well motivated. Its proposed methodological implementation with learned queries and cross-attention over visual and textual tokens, followed by temporal classification, seems like an appropriate design. Even if generally there are important limitations to assuming a fixed panel of abnormalities, this signal is only used for additional supervision and is not necessarily detrimental for representation learning.
Critical weaknesses
One of my key concerns regards the suitability of the proposed temporal supervision scheme, which relies on trajectory-level progression labels. Trajectories are often non-monotonic, especially at the patient level (e.g. throughout a long ICU stay). This limitation is very briefly acknowledged in Appendix C.3 without further discussion. Why not supervise with visit-level progression labels, which seems perfectly compatible with the Mamba head? Relatedly, it is not clear what image representations are expected to gain from temporal supervision—can you guess from a single radiograph whether the patient is improving overall? Lastly, the authors provided virtually no detail about the labelling process itself (prompts, sampling settings, evaluation, handling of uncertain progression, etc.), making it impossible to assess or replicate.
My second major concern is that it is not clear what exactly is being demonstrated in the experiments. The authors have not specified what MedCTA representations are used in the downstream tasks: encoder global representations vs Mamba outputs? single- vs multi-visit? abnormality- vs patient-scope? etc. This is further disorienting because the inputs/outputs of the evaluation tasks are never fully stated. Moreover, the baselines were described at high level in Appendix A.3, but without mention of architectures, parameter counts, or training data, which can greatly confound the interpretation of results and may undermine methodological improvement claims. The dynamic phrase grounding evaluation comprises just three qualitative heatmap examples, with no attempt to quantify spatial localisation across a dataset or compare with other models. Finally, discussion of results for each experiment was generally quite shallow, with limited insight on how the method's components might have led to improved results on each task.
Questions for authors
- The assumption of monotonic progression at both abnormality- and patient-scope needs further justification, and its trade-offs should be discussed.
- Because the LLM parsing is listed as a core contribution, the process must be described in detail to allow judging its appropriateness. Also, please clarify whether the labelling itself was evaluated for correctness against human ground truth (not just downstream performance).
- Specify precisely how the MedCTA model interfaces with each of the downstream tasks (inputs/outputs).
- Add the missing details for the chosen baselines, discuss possible confounders, and contextualise the fairness of the reported comparisons. If the baselines were retrained by the authors with similar settings to MedCTA, this must be stated explicitly, as it would significantly strengthen the reported evidence.
- Elaborate on the discussion of experimental findings: which design choices improved results and why (also contextualising with the ablations)? how does it compare to the baselines, and are there any patterns/trends amongst them? what are limitations of the proposed method? etc.
9 comments
Response to W1, Q5:
(3) Trajectory-level and Visit-level Progression Labels
In clinical practice, understanding both disease-specific evolution and overall patient disease progression is important for longitudinal clinical assessment. Visit-level supervision, which focuses on a single visit or changes between adjacent visits, primarily captures local changes and cannot directly constrain the overall direction of evolution across the entire patient observation window.
To address this issue, MedCTA proposes an LLM-driven joint trajectory learning framework. Through disease-level and patient-level trend supervision, it jointly learns disease-specific evolution representations and overall patient progression representations, thereby extending local visit-level change modeling to comprehensive clinical trajectory learning across the entire observation window.
To evaluate the contributions of complete sequence information and explicit temporal modeling to pre-trained representations, we introduce two comparison variants: First–Last and Mean Pooling. While keeping the training data, both types of trend labels, and supervision losses consistent, First–Last feeds only the first and last visits into the temporal branch, with both image and report sequence lengths set to 2. Mean Pooling retains the complete sequence, replaces the temporal encoder with average pooling along the temporal dimension, and performs the same trend prediction.
As shown in Tables 2 and 3, the complete MedCTA consistently outperforms both variants across all metrics. Compared with First–Last, MedCTA improves zero-shot ACC by 3.87 and 3.25 percentage points on COVIDx and RSNA, respectively, and improves linear-probe ACC by 3.26 and 0.62 percentage points, respectively. These results indicate that intermediate visit information contributes to learning more comprehensive longitudinal clinical representations. Furthermore, the complete MedCTA outperforms Mean Pooling, further demonstrating the contribution of explicit temporal modeling over simple temporal average aggregation. We have added the First–Last and Mean Pooling baselines to Section 4.3 of the revised manuscript.
Table 2. Zero-shot classification of pre-training variants on COVIDx and RSNA Pneumonia (%).
| Method | ACC (COVID) | F1 (COVID) | AUROC (COVID) | ACC (RSNA) | F1 (RSNA) | AUROC (RSNA) |
|---|---|---|---|---|---|---|
| first_last | 74.15 | 60.45 | 86.85 | 64.97 | 55.11 | 85.60 |
| mean_pooling | 76.76 | 62.84 | 87.42 | 66.51 | 56.37 | 87.49 |
| MedCTA | 78.02 | 63.93 | 89.62 | 68.22 | 57.91 | 88.60 |
Table 3. Linear-probe classification of pre-training variants on COVIDx and RSNA Pneumonia (%).
| Method | ACC (COVID) | F1 (COVID) | AUROC (COVID) | ACC (RSNA) | F1 (RSNA) | AUROC (RSNA) |
|---|---|---|---|---|---|---|
| first_last | 92.40 | 83.37 | 96.63 | 85.43 | 77.69 | 89.65 |
| mean_pooling | 93.54 | 84.61 | 95.52 | 85.29 | 77.53 | 89.57 |
| MedCTA | 95.66 | 88.48 | 96.92 | 86.05 | 79.25 | 89.82 |
Response to W2, Q3, Q4, Q5:
(1) Details of Downstream Tasks
- Temporal Image Classification: The pre-trained image encoder is frozen, and only the classification head is trained on Chest ImaGenome. The current and previous chest X-rays are processed by the shared encoder to generate image-level features, which are concatenated in the order of "current, previous" to predict the corresponding abnormality trend, including improvement, stability, or worsening. During MS-CXR-T testing, both the image encoder and classification head are frozen. This task uses image-level features.
- Temporal Sentence Similarity Classification: Two text sentences are separately encoded by the frozen text encoder to extract global features. After L2 normalization, cosine similarity is calculated, and ACC and AUROC are used for evaluation.
- Phrase Grounding: The image and text encoders are frozen. Similarities between image patch features and text features are used to generate heatmaps, and IoU, Dice, and CNR are reported in conjunction with manually annotated bounding boxes. As shown in Table 4, MedCTA achieves higher values than TAMM on all three metrics, with the main improvement observed in CNR.
Table 4. Quantitative phrase-grounding results.
| Method | IoU | Dice | CNR |
|---|---|---|---|
| TAMM | 0.1108 | 0.1894 | 0.4043 |
| MedCTA | 0.1124 | 0.1917 | 0.5895 |
- Image Classification: In the original RSNA/COVIDx experiments, the data were split into training, validation, and test sets at a ratio of 80%/10%/10%. The image encoder was frozen, and the linear classifier was fine-tuned. Therefore, this task is Linear Probe. In the newly added zero-shot experiments, both the image and text encoders are frozen, and classification is performed based on the similarity between images and class texts, without training a classifier using target labels. As shown in Table 5, MedCTA outperforms TAMM and ALTA on ACC, F1, and AUROC across both datasets.
Table 5. Zero-shot image–text classification on COVIDx and RSNA Pneumonia (%).
| Method | ACC (COVID) | F1 (COVID) | AUROC (COVID) | ACC (RSNA) | F1 (RSNA) | AUROC (RSNA) |
|---|---|---|---|---|---|---|
| ALTA | 62.70 | 49.03 | 85.95 | 65.40 | 56.69 | 82.13 |
| TAMM | 76.12 | 59.19 | 86.15 | 66.85 | 56.19 | 81.62 |
| MedCTA | 78.02 | 63.93 | 89.62 | 68.22 | 57.91 | 88.60 |
(2) Experimental Data Splits
When constructing the MedCTA temporal pre-training dataset, we excluded downstream patients based on patient IDs (subject_id). MS-CXR-T, MIMIC-5×200, and the Chest ImaGenome subset used in our experiments involve 800, 957, and 16,001 patients, respectively. We took the union of the patient sets from these three datasets and excluded all images and reports belonging to these patients from MIMIC-CXR before constructing longitudinal sequences. The resulting pre-training dataset contains 46,564 patients and 61,164 sequences. We have added details of the data split to Appendix A.1 of the revised manuscript.
(3) Baseline Details
We categorize the baseline results according to their sources:
- Quoted results: ACC for temporal image classification, ACC for temporal sentence similarity classification, cross-modal retrieval metrics, and ACC and F1 for RSNA/COVIDx linear probing are quoted from the TAMM paper.
- Reproduced results: AUROC for temporal sentence similarity classification and RSNA/COVIDx classification. Specifically, TAMM was retrained following its original method on the same patient-excluded data as MedCTA, while the remaining baselines were evaluated using their official pre-trained weights.
MedCTA follows the initialization strategy of TAMM, using publicly available MGCA weights for initialization, followed by temporal pre-training on the patient-excluded dataset. The initialization of the remaining baselines follows the experimental configurations reported in the TAMM paper. The parameter counts of all models are reported in Table 6. We have added the baseline details to Appendix A.3 of the revised manuscript.
Table 6. Model parameter counts (M)
| Method | Parameters (M) |
|---|---|
| MGCA | 172.73 |
| MedCLIP | 133.26 |
| PRIOR | 159.92 |
| MAVL | 170.65 |
| CARZero | 220.77 |
| Med-ST | 233.76 |
| ALTA | 265.65 |
| TAMM | 321.91 |
| MedCTA | 198.70 |
Response to W1, Q5:
(1) Non-monotonic Temporal Supervision
MedCTA learns the dominant direction of change over the entire patient observation window, allowing disease progression to exhibit non-monotonic changes across different visits. Each pre-training sequence contains up to four consecutive visits. Trend annotations are determined based on descriptions explicitly supported by clinical reports, following the priority of magnitude of change, persistence of change, and overall changes in major abnormalities to determine the overall trend within the observation window. We have added the prompt template for extracting temporal trend labels to the revised manuscript. During pre-training, image and text features from multiple visits are extracted through shared encoders and are used to construct patient-level and abnormality-level trajectory sequences, respectively, which are then temporally modeled using Mamba. Subsequently, patient-level and disease-level trend labels are used to supervise the corresponding trajectory representations. Through this temporal supervision, MedCTA learns longitudinal information about both overall patient disease progression and disease-specific changes.
To further evaluate whether the representations learned through the aforementioned trend supervision can transfer to multi-disease longitudinal image understanding tasks, we conduct an exploratory evaluation on the Global Trajectory Summarization multi-abnormality subset of MI-CXR [1]. Each sample contains five temporally ordered chest X-rays, one question, and five candidate longitudinal summaries. The task is to select the summary that best matches the evolution of multiple abnormalities. The data were split into training, validation, and test sets at the patient level, with proportions of 80%, 10%, and 10%, respectively. The image and text encoders were frozen to extract features for the five images and five candidate answers. The image features were average-pooled and mapped through a trainable projection head, followed by cosine similarity computation with the candidate text features. During training, the index of the correct answer was used as supervision, and only the projection head was optimized using a five-candidate cross-entropy loss. During testing, answers were selected and ranked according to their similarity scores.
As shown in Table 1, MedCTA achieves an ACC of 63.30% and an MRR of 79.03%, improving over TAMM by 1.50 and 1.92 percentage points, respectively. F1 and AUROC are also higher. The improvements in ACC and MRR jointly demonstrate advantages in both correct summary selection and ranking, providing exploratory evidence of the model's ability to represent longitudinal disease evolution. We have added the global trajectory summary matching task to Section 4.2.1 of the revised manuscript.
Table 1. Candidate-answer matching on the MI-CXR GTS multi-abnormality set (%).
[1] MI-CXR: A Benchmark for Longitudinal Reasoning over Multi-Interval Chest X-rays. ACL 2026.
(2) LLM Annotation Details
The offline Qwen2-7B-Instruct model processes complete reports arranged in chronological order and generates trend labels for the patient level and twelve abnormality categories: improvement
1, stability0, and worsening-1. The abnormality loss is computed only for valid labels, while non-monotonic processes are summarized according to the aforementioned dominant-trend criteria.To further assess the clinical correctness of the trend labels generated by the LLM, we conducted a manual annotation-based evaluation. We randomly sampled 100 longitudinal patient sequences, which were independently annotated by two authors. The average agreement rate between LLM-generated and human-annotated patient-level trend labels was 76%, while the inter-annotator agreement rate was 72%. For disease-level trend labels, the corresponding agreement rates were 69% and 73%, respectively. The agreement between the LLM and human annotations is comparable to the inter-annotator agreement, providing preliminary quantitative support for label quality.
Compared with patient-level trend labeling, disease-level trend recognition requires not only distinguishing among different disease types but also independently determining whether each disease is improving, stable, or worsening. This involves finer-grained semantic understanding and multidimensional judgment, and therefore entails greater annotation complexity. We have added the LLM annotation details to Appendix C.4 of the revised manuscript.