acceptodds
Withdrawn from ICLR 2027

Learning Temporally Equivariant Multi-Modal Large Language Models for Longitudinal Medical Image Comparison

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Longitudinal medical image comparison requires models to reason about how findings change over time. A reliable model should therefore behave consistently under temporal reversal: if a finding worsens from a historical study to a current study, reversing the image order should yield the corresponding inverse relation. We refer to this property as temporal equivariance. A controlled study on CheXTemporal shows that current general and medical multi-modal large language models (MLLMs) often violate this property: similar forward and reverse accuracies do not imply reliable bidirectional correctness. We propose TEMA, a Temporally Equivariant MLLM Adaptation framework for longitudinal medical image comparison. TEMA couples supervision on original and temporally reversed image pairs with a visual evidence consistency objective: temporal semantics should transform under reversal, while their supporting visual evidence should remain consistent. To account for different output structures, TEMA operates at different weighting granularities: sample-level weighting for closed-ended temporal VQA and temporal-token weighting for open-ended report generation. We evaluate our TEMA on temporal VQA and open-ended difference report generation across multiple MLLM backbones. On temporal VQA, TEMA consistently improves both balanced accuracy and strict temporal equivariance over standard LoRA fine-tuning and temporal-inversion baselines across all four evaluated backbones. On difference report generation, TEMA also improves clinically oriented generation metrics across backbones and remains competitive with specialized longitudinal report-generation baseline.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Temporally Equivariance, Longitudinal Medical Image Comparison, Multi-Modal Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TaScOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper suggests a new framework for recognizing temporal change in multi-modal large language models (MLLMs) with adaptation, called TEMA. Following the previous work, TILA, they adapted the basic concept which is the model's confidence or outputs are logically correct when the historical views are reversed to the MLLMs with adding visual evidence consistency and task-adaptive temporal weighting. They showed TEMA can improve the performance of MLLMs for temporal change both in VQA and report generation tasks than other adaptation framework in CheXTemporal and TemMedBench.

Critical strengths
  • The paper is well written and easy to follow.
  • The authors evaluate the proposed method on VQA and report generation to demonstrate that its applicability is not limited to a single task.
Critical weaknesses
  • A central reference for this work, TILA, reports not only retrieval scores and macro accuracy but also the temporal entity matching (TEM) score introduced in BioVIL-T [1]. TEM utilized a broad range of temporal keywords when they calculated the score, including "persistent," "grow," "expand," "elevate," "constant," "unfolded," etc.) Thus, it seems well suited to capturing temporal changes in this paper. However, the authors do not use this metric and instead limit their evaluation to forward balanced accuracy, reverse balanced accuracy, and accuracy excluding stable cases.
  • On the CheXTemporal gold set, the proposed method shows lower accuracy when stable cases are included (both FB Acc and RB Acc are lower than their respective counterparts excluding stable cases.) This suggests that the method may classify unchanged cases as changed. Such result appears inconsistent with both clinical needs and the paper's claims.
  • All results are based on internal evaluations. For temporal VQA, the authors train on the CheXTemporal training data and evaluate on its test data. For report generation, they partition the TemMed-Bench data for training and validation and evaluate only on a held-out test set from the same dataset.
  • The paper also lacks a comparison with ALTA [2], a prior study that focuses on temporal changes in medical vision-language alignment which is the similar approach with this paper.
  • The paper does not include any qualitative results to understand the work and check the results.

[1] Bannur, Shruthi, et al. "Learning to exploit temporal structure for biomedical vision-language processing." 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2023.
[2] Lian, Chenyu, et al. "Efficient medical vision-language alignment through adapting masked vision models." IEEE Transactions on Medical Imaging 44.11 (2025): 4499-4510.

Questions for authors
  • At the beginning of Section 3.3, the authors explain that Visual Evidence Consistency (VEC) is introduced to make image use observable, since reverse supervision alone does not establish whether the model appropriately uses visual evidence. However, they also caution that VEC should not be interpreted as faithful clinical grounding. Although attention maps or Grad-CAM would not establish faithful clinical grounding either, it would still be useful to determine whether the model attends to, interprets, and uses image regions near the changing lesions. (1) Are there any qualitative evaluations or analyses addressing this? (2) Does the method also work with general-domain models such as Qwen3-VL which is not pretrained on medical imagining?
  • How do the authors interpret the differences between accuracy with and without stable cases? What the model's performance specifically on stable cases?
Reviewer_hacROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TEMA, a LoRA-based framework for temporal equivariance in longitudinal medical image comparison, combining forward/reverse supervision, attention-based visual consistency, and task-specific weighting. Across four MLLM backbones on CheXTemporal and TemMed-Bench, TEMA improves directional VQA accuracy, joint forward–reverse correctness, and report-generation metrics over the evaluated baselines. Ablations and a human audit of reverse targets provide supporting evidence, though direct quantitative evaluation of temporal equivariance is limited to VQA.

Critical strengths

1. A well-motivated structural test of longitudinal reasoning. Temporal reversal provides a controlled way to test whether predictions respect the direction of change. Distinguishing simple consistency from joint correctness avoids rewarding models that produce mutually compatible but incorrect answers. This is a consequential contribution because conventional single-direction evaluation can conceal this failure mode.

2. Consistent empirical gains across multiple backbones under reasonably controlled comparisons. Non-Stable strict equivariance improves substantially across all four backbones, while report-generation subset-macro metrics also improve over standard LoRA. Matching optimizer updates and processed directional inputs reduces the likelihood that the gains merely reflect additional training. Together, these results support the practical usefulness of the training recipe beyond one model.

Critical weaknesses

1. The reported gains conceal a substantial deterioration in Stable-class recognition. From the stated macro-averaged accuracy definition, Stable recall can be recovered as (5,\mathrm{FBAcc}-4,\mathrm{FBAcc}_{w/o,Stable}). For Lingshu-7B, Table 1 therefore implies a decrease from approximately 70.92% with LoRA to 18.51% with TEMA; similarly large decreases occur for the other backbones. Meanwhile, overall forward balanced accuracy improves relatively modestly. This does not invalidate the directional gains, but it materially changes their interpretation: the method trades substantial stability recognition for improved change recognition. Without explicit per-class results and discussion, the evidence is insufficient to establish a broadly improved longitudinal comparator.

2. evidence isolating the framework’s additional contribution beyond reweighting and reverse supervision is limited. In Table 3, adding task-adaptive weighting raises strict equivariance from 14.83% to 33.19%, whereas VEC increases it further to 34.60%. The forward-only reweighting control is useful and supports a contribution from reversal, but it removes Stable supervision entirely, unlike TEMA’s retained forward Stable supervision. Moreover, component ablations are restricted to one backbone per task, with no reported run-to-run uncertainty. Consequently, the combined recipe is better supported than a robust, independently established benefit from visual evidence regularization. This matters because VEC is a principal methodological distinction from temporal-inversion training.

Questions for authors
  1. Can you confirm the Stable recalls implied by Table 1 and provide forward/reverse per-class recalls and confusion matrices? How should the large Stable regression be reconciled with the stated purpose of retaining full forward supervision for Stable examples?

  2. Can you provide a forward-only reweighting control that retains Stable supervision and matches the effective class weighting as closely as possible? Additionally, does the VEC improvement over reverse supervision plus task-adaptive weighting persist across repeated runs or another backbone?

  3. Is Strict Equivariance Accuracy w/o Stable computed as a sample-level average or a macro average over the four directional classes? Please provide its exact definition and, if it is sample-averaged, the corresponding macro-averaged result, since the non-Stable classes also have substantially different frequencies.

Reviewer_NyFKOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work identifies temporal equivariance in medical multimodal LLMs for longitudinal chest‑X‑ray comparison. When swapping historical‑current image pairs, existing models fail to consistently invert temporal clinical relations. In this purpose, this paper proposes the TEMA framework combining bidirectional reversed‑pair supervision, visual‑evidence consistency regularization, and task‑adaptive weighting for longitudinal medical image comparison. Evaluated on CheXTemporal across four MLLM backbones, TEMA shows effective.

Critical strengths
  1. The paper finds a Temporally Equivariant proporty in MLLM for longitudinal medical image comparison, and proposes a TEMA method to improve current works.
  2. TEMA works with off‑the‑pretrained multimodal LLMs through LoRA fine‑tuning. It does not require rebuilding heavy model architectures, showing good practicability for medical research communities.
  3. Beyond simple output‑level reverse supervision, it introduces visual‑evidence consistency loss on decoder attention maps. Task‑adaptive weighting further addresses the Stable‑class shortcut for VQA and highlights temporal tokens for report generation.
Critical weaknesses
  1. The TEMA framework is designed to pairwise two‑image chest‑X‑ray inputs. Real clinical practice contains multi‑visit sequential examinations. How to generalize temporal equivariance beyond two‑image pairs is a problem should be exploited.
  2. The visual‑evidence consistency builds upon decoder attention as visual evidence proxy, which may not serve as faithful clinical grounding, thus undermining the interpretability promise of this regularization term.
  3. The hyperparameters are tuned on one backbone and directly transferred to other models. There is insufficient head‑to‑head comparison against state‑of‑art specialized longitudinal radiology report generators.
Questions for authors

Please see the Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.