acceptodds
Under review as a conference paper at ICLR 2027

Longitudinal Radiology Report Generation via Progression Alignment and Pathology Semantic Learning

Abstract

Longitudinal radiology report generation requires representations that capture both clinical findings and their changes across examinations. To address these complementary requirements, we propose a unified framework for cross-modal progression alignment and pathology semantic enhancement. We represent progression as the residual between current and prior representations within each modality. Through temporal direction alignment, we encourage consistent directions of change between visual and textual representations, while auxiliary static alignment maintains image–report consistency within each examination. Cross-modal consistency alone, however, does not explicitly associate visual features with clinical findings. To establish these associations, a label-induced inverse optimal transport objective jointly learns pathology prototypes and a visual adapter, providing semantic guidance to the features used for generation. The resulting representations combine current visual evidence with prior image–report context to condition a frozen language model. Experiments on Longitudinal-MIMIC demonstrate strong performance in report generation and clinical agreement, with ablation studies and representation analyses supporting the complementary roles of progression alignment and explicit pathology semantic supervision.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Chest radiology report generation, Multimodal fusion, Medical artificial intelligence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_GFs6OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PAPE, a framework for longitudinal radiology report generation that combines progression alignment with pathology semantic learning. It models clinical change using the residuals between current and prior image and report representations, and aligns their progression directions while maintaining image–report consistency at each examination. The method also learns pathology prototypes from clinical observation labels through an inverse optimal transport objective, providing explicit semantic supervision for the visual features. These representations are fused with prior image–report context and used to condition a frozen language model for report generation. Experiments on Longitudinal-MIMIC show competitive performance on both language-generation and clinical efficacy metrics.

Critical strengths

The paper gives a clear formulation of longitudinal change by aligning the progression of image and report representations across examinations. This fits the task well and provides a direct way to model temporal clinical information.

Critical weaknesses
  1. In direction-alignment loss, all non-matching pairs in a batch are used as negatives. Patients can share similar clinical changes, so clinically related progression patterns may be pushed apart.
  2. Progression is defined by the difference between global image representations. This residual can also reflect changes in view, positioning, or acquisition, which are not separated from true clinical change.
  3. For studies with multiple images, features are averaged at corresponding token positions across views. Frontal and lateral radiographs have no direct spatial correspondence, making this averaging operation questionable before temporal modeling.
Questions for authors
  1. Does direction-alignment loss consider cases where different patients have similar clinical progression?
  2. Can view, positioning, or acquisition differences also affect the residual direction?
4 comments
AuthorsOpenReview ↗

Response to Reviewer GFs6

Thank you for your careful reading of the direction loss and the aggregation across views. In response, we revised the manuscript to explain the role of textual residuals in training and to discuss the limitations of contrastive alignment and view averaging. These changes make the method easier to follow and point to concrete ways to improve it.

For W1. Similar changes across patients

Thank you for raising this point. We acknowledge that instance-level contrastive objectives can introduce false negatives: unmatched image–report pairs may share clinical findings, and different longitudinal examination pairs may exhibit similar progression patterns. Our direction alignment aims to capture finding-specific clinical changes, rather than only broad categories such as “improving” or “worsening.” For example, improvement in pulmonary edema and improvement in pleural effusion involve different findings and need not represent the same semantic transition. Nevertheless, genuinely similar changes can occur across patients, and treating these pairs as negatives may penalize clinically meaningful similarities.

Alongside instance-level alignment, the label-guided prototype objective provides shared clinical semantic targets, while the report-generation objective supplies supervision from the full reports. These objectives provide complementary semantic constraints, although they do not explicitly eliminate false negatives. Our ablation results show that removing Ldir\mathcal{L}_{\mathrm{dir}} reduces both report-generation quality and clinical F1, supporting an overall benefit of progression alignment under the evaluated setting despite this limitation.

In future work, we plan to explore a semantics-aware soft alignment mechanism that adjusts the contribution of negative pairs according to their clinical similarity. For direction alignment, this could use label patterns across both prior and current examinations to identify potentially similar progression, assigning such pairs a reduced negative penalty. We will clarify this limitation and discuss this extension in the revised manuscript.

For W2 and Q2. Acquisition effects on residuals

Thank you for pointing this out. We agree that differences between global image representations can reflect changes in projection, positioning, or acquisition conditions, in addition to clinical progression. Following your suggestion, we revised the direction-alignment objective to use textual residuals as detached semantic references for learning visual residuals:

Ldir=C(Dv,sg⁡[Dt]).\mathcal{L}_{\mathrm{dir}} = \mathcal{C}\left(\mathcal{D}^{\mathrm v}, \operatorname{sg}[\mathcal{D}^{\mathrm t}]\right).

We retain both contrastive terms and stop gradients through the textual residuals in both terms. Consequently, Ldir\mathcal{L}_{\mathrm{dir}} updates the visual adapter, while the text adapter continues to receive supervision from the other training objectives. This revision encourages visual residuals to align with the clinical changes described in the reports, using the textual residuals as semantic guidance.

With the revised formulation, BLEU-4 increases from 0.134 to 0.136, METEOR from 0.171 to 0.174, and clinical F1 from 0.510 to 0.512. These results show improved report-generation quality and clinical agreement with the revised model. We have updated the methodological description and experimental results accordingly.

For W3. Aggregation across views

Thank you for pointing this out. We agree that frontal and lateral radiographs do not have direct anatomical correspondence at matching token positions. Our current averaging operation is a simple aggregation heuristic for obtaining a fixed-length examination representation; it does not establish spatial correspondence between views.

We would also like to clarify that the progression-direction loss operates on examination-level global representations obtained by averaging the global tokens from individual images. It therefore does not compute temporal differences between spatially matched patch tokens. However, the full token sequences used for report generation are also averaged across views before semantic compression and temporal fusion. We acknowledge that this operation can mix features from different anatomical regions and lose view-specific information.

We have already clarified the limitations in the appendix. A natural improvement is to preserve the token sequence from each view and let the Perceiver attend jointly to the concatenated sequences, avoiding position-wise averaging before temporal modeling. We plan to investigate this view-preserving aggregation strategy in future work.

AuthorsOpenReview ↗

Supplementary response to Reviewer GFs6

For Q1. Similar clinical progression

Thank you for raising this point. We acknowledge that instance-level contrastive objectives can introduce false negatives: unmatched image–report pairs may share clinical findings, and different longitudinal examination pairs may exhibit similar progression patterns. Our direction alignment aims to capture finding-specific clinical changes, rather than only broad categories such as “improving” or “worsening.” For example, improvement in pulmonary edema and improvement in pleural effusion involve different findings and need not represent the same semantic transition. Nevertheless, genuinely similar changes can occur across patients, and treating these pairs as negatives may penalize clinically meaningful similarities.

Alongside instance-level alignment, the label-guided prototype objective provides shared clinical semantic targets, while the report-generation objective supplies supervision from the full reports. These objectives provide complementary semantic constraints, although they do not explicitly eliminate false negatives. Our ablation results show that removing Ldir\mathcal{L}_{\mathrm{dir}} reduces both report-generation quality and clinical F1, supporting an overall benefit of progression alignment under the evaluated setting despite this limitation.

In future work, we plan to explore a semantics-aware soft alignment mechanism that adjusts the contribution of negative pairs according to their clinical similarity. For direction alignment, this could use label patterns across both prior and current examinations to identify potentially similar progression, assigning such pairs a reduced negative penalty. We have clarified this limitation and discuss this extension in the revised Appendix.

Thanks for the explanations and updated results. The revised loss helps, but I still have concerns about false negatives and averaging across views. I’m keeping my original score.

AuthorsOpenReview ↗

Further clarification to Review GFs6

Thank you for considering our explanations and updated results. We appreciate your remaining concerns and would like to clarify them as follows:

For false negatives

We acknowledge that our current formulation does not explicitly correct false negatives. Our rationale for retaining it is that instance-level cross-modal contrastive learning provides useful supervision from observed correspondences, even when some unmatched samples may share semantic content. This training principle continues to be used in recent work, including but not limited to TIM [1], Sparse CLIP [2], CLIP-FMoE [3], LIVE [4], and TILA [5]. Although their objectives and sampling strategies differ, their published formulations retain unmatched cross-instance pairs as negatives without explicitly identifying or downweighting semantically equivalent cross-instance pairs.

In our task, the ablation results provide direct evidence of the overall benefit: removing the direction-alignment loss reduces BLEU-4 from 0.136 to 0.128 and clinical F1 from 0.512 to 0.480. These results support retaining progression alignment in the current framework, while leaving room for improvement in how negatives are handled. We therefore acknowledge false negatives as an unresolved limitation and plan to investigate a semantics-aware soft alignment strategy that adjusts negative-pair weights according to clinical label similarity across both prior and current examinations.

[1] TIM: Temporal Decoupling with Iterative Mutual-Refinement Model for Longitudinal Radiology Report Generation. CVPR, 2026.

[2] Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning. ICLR, 2026.

[3] CLIP-FMoE: Scalable CLIP via Fused Mixture-of-Experts with Enforced Specialization. ICLR, 2026.

[4] Language-Instructed Vision Embeddings for Controllable and Generalizable Perception. ICLR, 2026.

[5] Temporal Inversion for Learning Interval Change in Chest X-Rays. CVPR, 2026.

For multi-view averaging

Our experiments use only frontal chest radiographs, including anteroposterior (AP) and posteroanterior (PA) projections; lateral radiographs are excluded. This restriction was intended to reduce representation differences arising from mixing frontal and lateral projections, so that longitudinal residuals could focus more closely on clinical changes within frontal radiographs. Our earlier response did not make this restriction sufficiently clear. When multiple eligible images are available for an examination, averaging is performed only among these frontal images. The resulting global token supports the auxiliary alignment objectives, while the full aggregated sequence is used for report generation.

Thus, the specific concern about averaging frontal and lateral anatomical layouts does not apply to our experiments. Nevertheless, we agree that differences in AP/PA projection and positioning can still affect spatial correspondence, and averaging remains a simple aggregation strategy. We have clarified the frontal-only input setting and its motivation in the implementation details and revise the limitations accordingly.

For both prior and current examinations, we use only frontal chest radiographs acquired in anteroposterior (AP) or posteroanterior (PA) projections and lateral radiographs are excluded. This restriction aims to reduce representation differences arising from mixing frontal and lateral projections, allowing longitudinal residuals to focus more closely on clinical changes within frontal radiographs. When an examination contains multiple eligible frontal images, their adapted token sequences are averaged at the same sequence indices to obtain an examination-level representation.

Thanks for you feedback again! If you have any further concerns, we would be happy to provide clarification and improve the manuscript based on your feedback.

Reviewer_1xxzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets longitudinal radiology report generation, which leverages current chest X‑ray plus prior examination data to generate radiology reports. The proposed PAPE framework introduces two auxiliary losses on top of a frozen LLaMA2‑7B: 1) cross‑modal progression alignment, aligning the direction of vision‑text temporal residuals plus static per‑examination image‑text alignment; 2) label‑driven inverse optimal transport (IOT) to learn pathology prototypes and ground visual features to clinical labels. Perceiver and temporal cross‑attention fuse multi‑modal history‑current information for LLM generation. Evaluated on Longitudinal‑MIMIC, PAPE obtains competitive NLG metrics and strong clinical F1. Ablations and representation analyses validate the effectiveness of each component.

Critical strengths
  1. Well-motivated design. The method addresses both current findings and changes from the previous examination, which are important for longitudinal reporting.
  2. Rigorous analysis: besides end‑to‑end metrics, it provides representation‑level diagnostics and qualitative case study; ablations prove complementary effects of three losses.
Critical weaknesses
  1. Evidence of superiority is limited. Evaluation covers only one dataset, and some differences are small: BLEU-4 is 0.134, compared with 0.133 for STREAM and MARE, while clinical F1 is slightly below TIM-II. Confidence intervals or repeated runs are not reported.
  2. Lacks radiologist human evaluation; clinical assessment fully depends on CheXbert automatic metric.
  3. Pathology prototypes are bounded to fixed 14 coarse MIMIC‑CXR labels, cannot handle unseen disease categories.
Questions for authors
  1. How would PAPE generalize to other modalities like CT/MRI? Do pathology prototypes need retraining?
  2. Can you add small‑scale radiologist human evaluation for factual correctness and hallucination?
  3. Why does linear‑probing performance barely improve with IOT, while prototype‑based recognition improves a lot?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 1xxz

Thank you for your careful review, especially your comments on variability and clinical evaluation. We revised the manuscript to clarify the scope of the evidence. The repeated runs and radiologist assessment below help put our findings in context.

For W1. Experimental variability and evaluation scope

Thank you for this constructive comment. We agree that small differences in individual metrics should be interpreted cautiously. To assess experimental variability, we conducted repeated experiments with three random seeds for our method. The standard deviations across these runs are provided below:

MetricBLEU-1BLEU-2BLEU-3BLEU-4MTRR-LCE-PCE-RCE-F1
SD0.0090.0030.0020.0010.0030.0100.0050.0030.003

Following the methodological revisions described in our other responses, the updated model achieves 0.136 BLEU-4, compared with 0.133 for STREAM and MARE, and 0.512 clinical F1, comparable to the 0.511 reported for TIM-II. These results indicate competitive performance across report-generation and clinical metrics.

Our controlled ablations provide additional evidence for the proposed design. Under the matched experimental setting, replacing IOT with sigmoid-based multilabel BCE reduces BLEU-4 from 0.136 to 0.132 and clinical F1 from 0.512 to 0.503.

Longitudinal-MIMIC is a large-scale, widely adopted benchmark specifically constructed for longitudinal radiology report generation. It contains longitudinal records from 26,625 patients, with paired examinations and corresponding reports that directly match our task setting. Our evaluation setting is consistent with representative longitudinal report-generation methods, including Prefilling, MARE, TIM, BiOTPrompt, and Diff-RRG, whose reported experiments are also conducted exclusively on Longitudinal-MIMIC. This large-scale benchmark provides paired prior and current examinations with corresponding reports, directly matching our task setting. We follow its official split to facilitate comparison with existing methods, while acknowledging that evaluation on additional datasets would further strengthen the evidence for generalization.

For W2 and Q2. Radiologist evaluation

Thank you for this valuable suggestion. To further assess the factual correctness and hallucination of the generated reports, we conducted a small-scale radiologist evaluation. Due to time constraint, we randomly sampled 20 longitudinal cases from the test set and invited three radiologists to independently evaluate the reports generated by PAPE.

The radiologists assessed each generated report using a four-level rating scale based on the current image and the corresponding longitudinal clinical context. A score of 4 indicates that the report is factually accurate and largely complete, without apparent hallucinations. A score of 3 indicates that the report is generally accurate but contains minor omissions or non-critical wording deviations. A score of 2 indicates that the report is only partially correct and contains noticeable omissions, factual errors, or inaccurate descriptions of clinical changes. A score of 1 indicates severe factual errors or clear hallucinations, with low overall clinical reliability. The three radiologists performed the evaluation independently, and we summarized their ratings across the sampled cases.

The generated reports obtained an average score of 2.6, with 24 of 60 ratings (40%) receiving a score of 3 or 4. These results suggest that the generated reports were accurate for a subset of cases, but factual errors, omissions, and inaccurate descriptions of clinical changes remain challenges.

AuthorsOpenReview ↗

Supplementary response to Reviewer 1xxz

For W3 and Q1. Prototype coverage and transfer to CT or MRI

Thank you for pointing out this limitation. Our current prototype supervision follows the standard 14-category annotation scheme provided with MIMIC-CXR. These categories originate from CheXpert and were selected based on their prevalence in radiology reports and clinical relevance (Irvin et al., 2019). They provide supervision for common chest radiographic observations, but do not constitute an exhaustive disease taxonomy.

The prototypes serve as auxiliary supervision for the visual adapter, while report generation is supervised using the full free-text reports. Therefore, the 14-category prototype set does not impose a fixed output vocabulary on the generated reports. We nevertheless acknowledge that the current prototype module does not explicitly represent categories outside this set. The IOT formulation supports a general number of prototypes KK, allowing additional categories to be incorporated and learned when suitable annotations are available. We will clarify this distinction and acknowledge the limited coverage of the current prototype supervision.

Thanks for the meaningful questions! The progression alignment and pathology prototype learning components of PAPE are not restricted to chest radiographs in principle and could be extended to other modalities, such as CT or MRI, given longitudinal images, reports, and corresponding clinical labels. However, because visual characteristics and clinical observation categories may differ across modalities, such an extension would require a suitable visual encoder and adaptation of components such as the visual adapter.

Regarding the pathology prototypes, if the target task uses the same observation categories, the existing prototypes could potentially be used for initialization and then fine-tuned on data from the target modality. If the observation categories differ, prototypes for the new categories would need to be defined and learned. Thus, whether the prototypes need to be trained from scratch depends on the target modality and label set.

Our current experiments are limited to longitudinal chest radiographs, and we have not evaluated PAPE on CT or MRI. We therefore do not claim cross-modality generalization based on the present results.

For Q3. Linear probing and prototype recognition

As shown in Fig. 4(a), the performance of Baseline + Linear Probe is already close to that of the full model. This is because linear probing freezes the visual representations after model training and trains an additional classifier to evaluate the linear decodability of clinical observation categories from the learned features. Therefore, this result reflects the classification capability of the existing representations under the specified linear-probing protocol, rather than the performance of direct prototype-based recognition. Since the Baseline representations already contain substantial information about disease categories, adding the auxiliary objectives does not lead to a substantial improvement in their linear separability. This is consistent with the relatively small performance differences among the variants in Fig. 4(a).

In contrast, as shown in Fig. 4(b), the Baseline, LstatL_{\mathrm{stat}}, and LdirL_{\mathrm{dir}} variants perform relatively poorly under direct prototype-based recognition. This suggests that the visual features learned by these variants do not yet form well-aligned relationships with the pathology prototypes. After introducing IOT-based prototype learning, prototype-based recognition improves substantially and approaches saturation under the current training and evaluation protocol. This improvement results from the fact that the IOT objective jointly optimizes the visual adapter and the pathology prototypes under observation-label supervision. Consequently, the cosine similarities between visual features and the corresponding prototypes become more closely aligned with the clinical observation labels.

Therefore, Fig. 4(a) and Fig. 4(b) measure two different capabilities. Linear probing evaluates whether clinical observation information can be decoded from the visual representations using an independently trained linear classifier, whereas prototype-based recognition directly evaluates the matching between visual features and the pathology prototypes learned by IOT. The main effect of IOT is reflected in the latter evaluation. It improves the feature–prototype geometry and provides the visual representations used for report generation with more explicit pathology-level semantic information.

AuthorsOpenReview ↗

Response to ethics concern for reviewer 1xxz

Ethics concern. Clinical risks of generated reports

Thank you for raising this concern. We agree that generated reports may miss abnormalities, describe findings that are not present, or misstate changes between examinations. Our small radiologist assessment in W2 and Q2 highlights these risks, but does not establish that PAPE is safe for clinical use.

We evaluate PAPE as a research system. Before any clinical use, further validation on independent cohorts would be needed to assess how often these errors occur and how serious they are. A radiologist would also need to verify each generated draft against the images and clinical history before it is used in patient care. We will make these limitations explicit in the revised manuscript.

Reviewer_SahZOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on longitudinal radiology report generation and proposes a unified framework for cross-modal progression alignment and pathology semantic enhancement. It aims to make different modalities (images and text) exhibit consistent directions of change by aligning the residuals between current and prior representations. Moreover, to strengthen the clinical basis, the authors propose a data-driven pathology semantic enhancement strategy. This strategy uses the 14 observation labels provided with MIMIC-CXR to supervise the learning of corresponding pathology prototypes through a label-induced inverse optimal transport (IOT) objective.

Critical strengths
  1. The paper is easy to follow, and its motivation is clear.

  2. The authors decouple the optimization problem for longitudinal report generation into direction and state alignment.

  3. The experiments and analyses in this paper are solid.

Critical weaknesses
  1. The evaluation should include additional clinically relevant metrics, such as GREEN and RaTEScore, to better assess the clinical quality of the generated reports.

  2. The temporal direction alignment and static alignment objectives (Eqs. 6–10) may introduce false negatives. Currently, the authors treat all mismatched image–text pairs within a batch as negatives. However, different examinations can share similar clinical findings, and different longitudinal pairs may exhibit similar directions of clinical change.

  3. In this work, the prior information used to condition current report generation includes both prior images and prior reports. However, some methods in Table 1 do not use prior reports as input, such as BiOTPrompt and HERGen. For STREAM, the authors should clarify whether the reported results use prior reports and ensure that the comparison uses matched input conditions. Its performance with prior reports is higher than that of the proposed method.

  4. The central claim concerns clinical progression, but the evaluation primarily relies on text-overlap metrics and observation-level clinical agreement. It does not directly assess the correctness of the generated descriptions of clinical progression.

  5. The residual directions are intended to capture clinical progression. However, representation differences may also arise from acquisition conditions (e.g., position, projection, or breathing state) or report wording, which do not necessarily reflect clinical changes. These factors may affect the reliability of the residual-based approach.

Questions for authors
  1. How are the 14 prototypes initialized?

  2. Since the prototypes correspond to 14 clinical observation categories, could the authors provide visualizations of the prototype features?

  3. The data-driven pathology semantic enhancement strategy relies on prototype matching. Have the authors compared this approach with an auxiliary classification task supervised directly by the observation labels? Please clarify the distinct contribution of the formulation and compare it with standard multi-label classification objectives.

  4. How are the loss weights lambda selected?

  5. Given that LLaMA2-7B is an older LLM, why did the authors choose it over more recent alternatives?

  6. LLaMA2-7B is a general-purpose LLM without medical specialization, and its parameters are frozen in this framework. Have the authors explored fine-tuning the LLM to improve its clinical capabilities?

8 comments
AuthorsOpenReview ↗

Response to Reviewer Sahz

Thank you for the detailed review. We carefully revised the manuscript in response to your comments, adding clinical and progression evaluations and clarifying the comparison settings and the role of textual residuals in training. Your feedback helped us make the evaluation more complete and the method easier to assess.

For W1. GREEN and RaTEScore

Thank you for the suggestion. To further assess the clinical quality of the generated reports, we additionally computed GREEN and RaTEScore. Due to time and computational constraints, we reproduced a selection of recent representative methods including R2GenGPT, BiOTPrompt and MARE. The results are shown below:

MethodGREEN ↑RaTEScore ↑
R2GenGPT0.24910.5030
BiOTPrompt0.28620.5130
MARE0.29470.5270
Ours0.31830.5440

As shown in the table, our method achieves the best performance on both GREEN and RaTEScore, obtaining scores of 0.3183 and 0.5440, respectively. Compared with the strongest competing method, MARE, our method improves GREEN by 0.0236 and RaTEScore by 0.0170. These results further support the effectiveness of PAPE in improving the clinical consistency and factual quality of generated reports. We have included these additional metrics and results in the revised Appendix.

For W2. False negatives

Thank you for raising this point. We acknowledge that instance-level contrastive objectives can introduce false negatives: unmatched image–report pairs may share clinical findings, and different longitudinal examination pairs may exhibit similar progression patterns. Our direction alignment aims to capture finding-specific clinical changes, rather than only broad categories such as “improving” or “worsening.” For example, improvement in pulmonary edema and improvement in pleural effusion involve different findings and need not represent the same semantic transition. Nevertheless, genuinely similar changes can occur across patients, and treating these pairs as negatives may penalize clinically meaningful similarities.

Alongside instance-level alignment, the label-guided prototype objective provides shared clinical semantic targets, while the report-generation objective supplies supervision from the full reports. These objectives provide complementary semantic constraints, although they do not explicitly eliminate false negatives. Our ablation results show that removing Ldir\mathcal{L}_{\mathrm{dir}} reduces both report-generation quality and clinical F1, supporting an overall benefit of progression alignment under the evaluated setting despite this limitation.

In future work, we plan to explore a semantics-aware soft alignment mechanism that adjusts the contribution of negative pairs according to their clinical similarity. For direction alignment, this could use label patterns across both prior and current examinations to identify potentially similar progression, assigning such pairs a reduced negative penalty. We have clarified this limitation and discuss this extension in the revised Appendix.

For W3. Prior reports and STREAM

Thank you for pointing out that the methods in Table 1 differ in the information they receive at inference. We agree that prior reports can provide explicit descriptions of earlier findings and that access to this information matters when interpreting the results. Our method uses both prior images and prior reports. BiOTPrompt and HERGen do not use prior reports, while Prefilling, Diff-RRG, MARE, and TIM do. We have documented the inference inputs for each configuration in the Appendix, with particular attention to whether prior reports are provided.

Table 1 lists STREAM‡ in the longitudinal group. This setting uses one image per examination and incorporates prior images when available, without prior reports. Appendix A.5 discusses STREAM*, which uses multiple views and a prior report. We report the published STREAM scores for context, as the comparisons with PAPE do not hold the historical inputs and view configurations fixed.

We agree with the reviewer that STREAM* reports higher BLEU-1 through BLEU-4 and ROUGE-L scores than our method. Its reported BLEU-4 and ROUGE-L scores are 0.139 and 0.297, compared with 0.136 and 0.291 for our method. STREAM* can use images from up to two prior examinations and supplements these inputs with regional descriptions obtained through a separate detection and retrieval pipeline. In STREAM’s own ablation, adding regional retrieval to its multiview temporal model increases BLEU-4 from 0.131 to 0.137 (Table IV), illustrating the benefit of this additional stage. Our method reaches numerically close scores using the current radiographs and a single prior examination with its report. Cross-modal alignment and pathology prototype supervision guide representation learning during training, and the model generates reports directly from these patient inputs at inference.

AuthorsOpenReview ↗

Supplementary response to Reviewer Sahz

For W4. Clinical progression evaluation

Thank you for this important comment. We agree that text-overlap metrics and observation-level clinical agreement alone cannot fully assess the correctness of clinical progression descriptions in the generated reports. To address this concern, we conducted an additional evaluation focused on clinical progression direction in Appendix A.8. Specifically, we extracted the clinical change direction from the generated reports and categorized it into three classes: Stable, Worsening, and Improving. We then calculated the F1-score for each category and the average F1-score.

Table 4: F1-score for Clinical Progression Classification

ModelStableWorseningImprovingAvg.
PAPE w/o Ldir\mathcal{L}_{\mathrm{dir}}0.9100.3390.6790.643
PAPE0.9130.3570.6950.655

As shown in the table, the full model consistently outperforms the variant without the direction-alignment loss Ldir\mathcal{L}_{\mathrm{dir}} across all three progression categories. The average F1-score improves from 0.643 to 0.655. In particular, the F1-scores for Stable, Worsening, and Improving increase by 0.003, 0.018, and 0.016, respectively.

Notably, the direction-alignment loss provides larger improvements for the Worsening and Improving categories, which involve explicit clinical changes, whereas the gain for the Stable category is relatively smaller. These results suggest that Ldir\mathcal{L}_{\mathrm{dir}} helps the model more accurately identify and describe the direction of clinical changes across longitudinal examinations.

For W5. Acquisition and wording differences

Thank you for highlighting this issue. We agree that representation differences may reflect acquisition conditions or report wording, in addition to clinical changes. Following your suggestion, we have revised the direction-alignment objective to use textual residuals as gradient-detached semantic references for learning visual residuals. Specifically, we retain both contrastive terms but stop gradients through the textual residuals in both terms. Consequently, Ldir\mathcal L_{\mathrm{dir}} updates the visual adapter, while the text adapter remains trainable through static alignment and report generation.

This modification uses the clinical semantics expressed in paired reports to guide visual residual learning, encouraging visual changes to agree with the reported clinical changes. We have clarified this motivation and the corresponding gradient flow in the revised method section. In our updated experiments, the revised model improves both report-generation quality and clinical efficacy metrics, supporting the benefit of this text-guided optimization strategy. Our main revisions are as follows:

In Section 2.1:

Longitudinal cross-modal progression alignment then encourages textual residual to serve as a gradient-detached semantic reference for learning the visual residual.

In Section 2.2:

Ldir=C(Dv,sg[Dt]).\mathcal L_{\mathrm{dir}}=\mathcal C(\mathcal D^{\mathrm v},\text{sg}[\mathcal D^{\mathrm t}]).

This objective encourages cross-modal agreement in progression direction for each examination pair while distinguishing it from other pairs in the batch. It should be noted that sg[⋅]\text{sg}[\cdot] denotes the stop-gradient operation so that gradients from Ldir\mathcal L_{\mathrm{dir}} only participate in the update of visual adapter, while the textual residual is treated as a detached semantic reference.

AuthorsOpenReview ↗

Supplementary response to Reviewer Sahz

For Q1. Prototype initialization

Thank you for asking about the initialization and optimization of the prototypes. We represent the 14 pathology prototypes as learnable vectors, with each prototype associated with a clinical observation category. Each vector is initialized by independently sampling its elements from a standard normal distribution. The prototypes therefore do not initially encode explicit pathological semantics. Representations are progressively learned through label supervision during training.

Specifically, the observation labels define a target assignment for each examination, while cosine similarities between the visual features and the prototypes determine the predicted assignment. The IOT objective encourages agreement between these assignments, providing a joint optimization signal for the visual adapter and the prototypes. The same set of prototypes is shared across prior and current examinations and receives supervision from both, allowing it to learn matching relationships associated with the clinical observation categories.

For Q2. Prototype visualizations

Thank you for this suggestion. We have added a “Visualization of Pathology Prototype Supervision” subsection before the Case Study. Figure 5 visualizes the spatial responses of visual features learned with prototype supervision using Grad-CAM. It presents five of the 14 clinical categories—Support Devices, Consolidation, Edema, Cardiomegaly, and Enlarged Cardiomediastinum—and compares the original radiographs, the variant without LIOT\mathcal{L}_{\mathrm{IOT}}, and the full model.

With IOT supervision, the maps highlight pulmonary opacities for Consolidation and Edema, cardiac or mediastinal regions for Cardiomegaly and Enlarged Cardiomediastinum, and visible device paths for Support Devices. In several examples, the variant without IOT shows prominent responses over peripheral soft tissues and subdiaphragmatic areas. These visualizations provide qualitative support that pathology prototype supervision helps the learned visual features focus on regions relevant to the corresponding clinical observations.

Reviewer_7cGROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose PAPE for longitudinal chest X-ray report generation, taking both the current scan and a previous image–report pair as input. The architecture uses trainable adapters, Perceiver compression, and temporal attention to condition a frozen LLaMA2-7B backbone. The main contribution lies in combining three auxiliary objectives: static image–text alignment, contrastive alignment between visual and textual temporal-residual directions, and clinical label supervision formulated via row-constrained inverse optimal transport. On Longitudinal-MIMIC, the authors provide baseline comparisons, ablations, and representation visualizations, where the full model improves clinical F1 from 0.471 to 0.510 over the vanilla baseline.

Critical strengths

Modeling both static state and temporal change makes intuitive sense. The framework combines standard image–report alignment at each time step with directional alignment on temporal residuals. Only aligning the direction (rather than the exact magnitude) of the residuals is a simple, sensible design choice, since visual and text features naturally scale differently across longitudinal scans.

Solid performance gains in a clean ablation setup. In Table 2, the authors keep the inputs and the generator backbone completely fixed while ablating the auxiliary losses. Adding the full set of auxiliary objectives boosts clinical F1 from 0.471 to 0.510 and BLEU-4 from 0.124 to 0.134. This clearly shows that the extra supervision—especially the clinical label guidance—is genuinely helpful in practice.

Critical weaknesses

The central claim about improved longitudinal understanding is not directly established. The evaluation relies on text overlap and current label accuracy, neither of which directly tests if the model can tell whether a condition is new, worsening, stable, improving, or resolved. Figure 3 is evaluated on training batches, and Figure 5 only gives a stable example. On top of that, Table 2 leaves out the most important ablation: LCE+Lstat+LIOTL_{\mathrm{CE}}+L_{\mathrm{stat}}+L_{\mathrm{IOT}} (the full model minus LdirL_{\mathrm{dir}}). Looking at the current numbers, LdirL_{\mathrm{dir}} alone only bumps clinical F1 from 0.471 to 0.475, and adding it on top of static alignment doesn't improve over static alignment alone. Without that missing ablation, we can't tell if the core direction loss is actually doing any heavy lifting in the full model.

The incremental contribution beyond existing alignment and label-supervision approaches is insufficiently demonstrated. As acknowledged in Section 2.3 and Appendix A.1, the row-constrained IOT objective is mathematically identical to cross-entropy with normalized positive targets. So beyond the theoretical framing, it doesn't really offer anything new over standard prototype classification. What's missing here is a fair comparison with a truly different standard loss, such as sigmoid-based multilabel BCE. Also, methods like HC-LLM and MARE already handle cross-modal temporal dynamics, and it's not obvious why direct residual-direction alignment is a better way to go. Even though the overall metrics improve, these gaps make it hard to judge how much the proposed designs actually matter.

Questions for authors

A sensible formulation for static states and temporal changes. Explicitly pairing static image–report alignment with temporal-residual direction alignment is a clean design. Focusing on residual direction allows the visual and text branches to have different feature magnitudes, which is a very reasonable way to handle cross-modal longitudinal pairs.

Clear empirical gains under controlled ablations. Table 2 evaluates the auxiliary losses while holding the input and generation pipeline constant. Under this controlled setup, the full model pushes clinical F1 from 0.471 to 0.510 and BLEU-4 from 0.124 to 0.134, showing that the combined auxiliary supervision (and particularly the clinical label loss) works well in practice.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 7cGR

Thank you very much for identifying the gaps in our progression evaluation and ablations. We revised the manuscript in response to your feedback, adding a progression assessment, two cases with interval change, and the requested full-model ablation. These additions provide clearer evidence for the role of direction alignment.

For W1. Clinical progression and the direction loss.

Thank you for raising this point. Our choice of metrics follows prior work on longitudinal radiology report generation. Prefilling, Diff-RRG, and MARE all report text-overlap and clinical efficacy metrics in their main evaluations on Longitudinal-MIMIC. We use these established metrics to compare generated reports with prior work in terms of textual overlap and agreement on current clinical findings.

We agree that longitudinal changes warrant a dedicated evaluation. Following TIM [6], we use changes in CheXbert observation labels between prior and current reports to derive three progression categories: Stable, Worsening, and Improving. We compare the transitions derived from the generated and reference reports and report per-category F1 and their mean (Appendix A.8, Table 4).

The full model achieves higher F1 in all three categories than the variant without Ldir\mathcal{L}_{\mathrm{dir}}, improving average F1 from 0.643 to 0.655. The gains are larger for Worsening (+0.018) and Improving (+0.016) than for Stable (+0.003). These results suggest that direction alignment improves agreement with the reference on the three evaluated categories. Separate scoring of new and resolved findings remains future work.

Table 4. F1 scores for clinical progression.

ModelStableWorseningImprovingAvg.
PAPE w/o Ldir\mathcal{L}_{\mathrm{dir}}0.9100.3390.6790.643
PAPE0.9130.3570.6950.655

To complement the stable case in Fig. 5 of the original submission, we added two qualitative examples showing worsening pleural effusion and improving pulmonary edema (Appendix A.7, Figure 7). For the highlighted finding in each case, the generated report agrees with the reference on the direction of change.

We have added the requested full-model ablation in Table 2. We remove only the direction-alignment loss, Ldir\mathcal{L}_{\mathrm{dir}}, while keeping the inputs, generation architecture, and other training objectives fixed. As the reviewer notes, adding the direction loss alone yields a modest clinical F1 gain from 0.471 to 0.475, and adding it to static alignment alone changes F1 from 0.485 to 0.484. In the full model, removing the direction loss reduces clinical F1 from 0.512 to 0.480 and BLEU-4 from 0.136 to 0.128. This ablation establishes the contribution of direction alignment to the full model’s report-generation performance. Whether it improves recognition of specific disease transitions remains a question for a dedicated evaluation on held-out cases.

[6]TIM: Temporal Decoupling with Iterative Mutual-Refinement Model for Longitudinal Radiology Report Generation. CVPR

For W2. Residual direction alignment

Thank you for raising this point. We agree that HC-LLM and MARE already model cross-modal temporal dynamics. HC-LLM aligns shared and time-specific representations through contrastive and geometric constraints, while MARE learns evolution relations and supervises their ability to connect historical and current features within each modality. Our design directly aligns the directions of the observed visual and textual changes over the same examination interval.

This design has a specific motivation. Taking the difference between current and prior representations cancels any shared additive component and emphasizes temporal change. Normalizing these residuals then focuses the alignment on their directions while allowing their magnitudes to differ. This is appropriate because image and text representations may encode the same clinical change at different scales. The contrastive objective further encourages each visual change to match the textual change from the corresponding examination pair. Together, these choices provide an explicit constraint on cross-modal agreement about what has changed.

To assess its contribution within the complete framework, we added the requested leave-one-out ablation. Removing Ldir\mathcal L_{\mathrm{dir}} reduces clinical F1 from 0.512 to 0.480, clinical recall from 0.470 to 0.424, and BLEU-4 from 0.136 to 0.128. Although direction alignment provides a modest gain when used alone, this ablation shows a larger contribution when combined with examination-level alignment and pathology supervision. Thesefore, we argue that these results support its complementary role in the full model.

AuthorsOpenReview ↗

Supplementary response to Reviewer 7cGR

For W2. IOT and sigmoid BCE

Thanks for your constructive suggestion. We agree that the row-constrained IOT objective reduces exactly to soft-target cross-entropy over prototype scores, up to a parameter-independent constant in the original KL objective. In our manuscript, the transport derivation explains the construction of the two distributions: t allocates equal mass to the observed positive labels, and q is a temperature-scaled softmax over cosine similarities to the learned prototypes. As we know, prototype learning admits different supervision schemes. For example, work [1] use a squared Euclidean penalty between a sample embedding and the mean of its positive-label prototypes.

Different from that, we formulate pathology prototype learning as a matching problem between examination-level visual features and clinical observation prototypes. The observed positive labels define a target assignment for each examination, while cosine similarities between its visual features and the prototypes define the costs that induce the predicted assignment. In this design, this formulation provides an explicit optimization principle for jointly learning the visual adapter and shared prototypes from label-derived matches across prior and current examinations.

We acknowledge that our previous description insufficiently explained this motivation. We have clarified the description in Section 2.3 as follows:

It can be observed that the core of the IOT-based bi-level optimization framework is to leverage known matches to guide the alignment between pathology prototypes and visual features.

Furthermore, when an examination has multiple positive labels, we give each corresponding prototype an equal share of the supervision, keeping the total weight per examination fixed. The model compares the visual features with all prototypes and learns to distribute the matching weight among the positive ones. Unlike this, BCE treats each label as a separate binary prediction task.

To evaluate the practical effect of this design, we conducted a matched BCE ablation by replacing the L_IOT, termed 'Full model w BCE'. The results are shown as follows:

MethodBLEU-1BLEU-2BLEU-3BLEU-4MTRR-LCE-PCE-RCE-F1
Baseline0.4090.2570.1740.1240.1600.2780.5260.4260.471
Baseline + L_stat0.4180.2610.1770.1280.1640.2810.5460.4370.485
Baseline + L_dir0.4210.2640.1800.1300.1650.2830.5390.4240.475
Baseline + L_IOT0.4170.2620.1790.1290.1640.2840.5490.4460.492
Baseline + L_stat+L_dir0.4250.2680.1830.1320.1660.2860.5500.4310.484
Baseline + L_stat+L_IOT0.4200.2630.1790.1280.1680.2790.5530.4240.480
Baseline + L_dir+L_IOT0.4220.2680.1840.1330.1670.2900.5830.4490.507
Full model w BCE0.4190.2660.1830.1320.1710.2870.5520.4610.503
Full model0.4290.2720.1860.1360.1740.2910.5630.4700.512

Replacing L_IOT with sigmoid-based multilabel BCE reduces performance across all six NLG metrics. For example, BLEU-4 decreases from 0.136 to 0.132, and ROUGE-L from 0.291 to 0.287. Clinical precision, recall, and F1 also decrease, with F1 dropping from 0.512 to 0.503. Besides, by comparing 'Baseline + L_stat+L_dir' and 'Full model w/ BCE', we can find that using BCE will lead to the performance in NLG metrice significant decline. We argue that the label-guided matching in L_IOT helps coordinate the joint optimization of pathology prototypes and the visual adapter, potentially reducing interference from insufficiently trained prototype representations during visual feature learning. Within this comparison, our IOT formulation provides more effective guidance for report generation than independent label-wise BCE supervision, offering empirical support for its practical value.

[1] Partial Multi-View Multi-Label Classification via Semantic Invariance Learning and Prototype Modeling, ICML 2024.

Comments on the paper

2 comments

Dear Reviewer,

As the discussion period is approaching its end, we would like to kindly follow up on our rebuttal and the additional clarifications we provided.

We would greatly appreciate it if you could let us know whether our responses have adequately addressed your concerns. If any questions remain, we would be happy to provide further clarification before the discussion closes.

Thank you again for your time and constructive feedback.

Authors

AuthorsOpenReview ↗

Summary of Rebuttals

We thank all reviewers and the AC for their time, constructive feedback, and valuable discussions. We have carefully responded to all concerns and substantially strengthened the manuscript through methodological revisions, additional experiments, and more comprehensive clinical evaluations. Below, we summarize our key contributions and responses to the main concerns.

Highlights of our work.

  • Explicit cross-modal progression alignment (7cGR, SahZ, GFs6). Unlike methods that implicitly model temporal relationships, PAPE directly aligns the directions of visual and textual changes across examinations. The newly added leave-one-out ablation demonstrates its contribution: clinical F1 improves from 0.480 to 0.512, and BLEU-4 from 0.128 to 0.136 when direction alignment is included in the full framework.

  • Pathology-guided semantic learning (7cGR, SahZ, 1xxz). PAPE jointly learns pathology prototypes and visual representations through label-guided matching, providing explicit clinical semantic supervision. Our newly added comparison with multilabel BCE confirms its practical benefit, improving clinical F1 from 0.503 to 0.512 and BLEU-4 from 0.132 to 0.136.

  • Strong report-generation and clinical performance (SahZ, 1xxz). The revised model achieves 0.512 clinical F1 and 0.136 BLEU-4 on Longitudinal-MIMIC. Additional evaluations demonstrate competitive clinical quality, with GREEN of 0.3183 and RaTEScore of 0.5440, outperforming the representative baselines evaluated using these metrics.

Main concerns and our responses.

  • Does progression alignment genuinely capture longitudinal clinical changes? (7cGR, SahZ). We introduced a dedicated progression evaluation based on Stable, Worsening, and Improving categories. Adding direction alignment improves average progression F1 from 0.643 to 0.655, including gains of 0.018 for Worsening and 0.016 for Improving. We also added qualitative examples illustrating both types of clinical changes. While Worsening remains challenging (F1 = 0.357), we analyzed possible causes, including missed abnormalities and differences in observation-label distributions between training and test sets. These results provide additional evidence of the benefits of progression alignment while identifying opportunities for further improvement.

  • Is the IOT formulation meaningfully different from standard classification supervision? (7cGR, SahZ). We explicitly clarified that the IOT's practical contribution lies in label-guided matching between visual features and learnable pathology prototypes. Under matched experimental conditions, replacing IOT with sigmoid-based multilabel BCE decreases BLEU-4 from 0.136 to 0.132 and clinical F1 from 0.512 to 0.503. Additional Grad-CAM visualizations illustrate improved responses in clinically relevant anatomical regions.

  • False negatives, acquisition variations, and multi-view aggregation (SahZ, GFs6). We revised the direction-alignment objective to use gradient-detached textual residuals as semantic references for visual learning. We also clarified that our experiments use frontal radiographs only (AP/PA), excluding lateral views; thus, the concern about averaging frontal and lateral anatomical representations does not apply to our experiments.

  • Clinical reliability, statistical variability, and evaluation scope (SahZ, 1xxz). Beyond conventional metrics, we added GREEN and RaTEScore comparisons, three-seed experiments (SD = 0.001 for BLEU-4 and 0.003 for clinical F1), and radiologist assessments. Three radiologists evaluated an initial 20 cases, followed by an additional 20 cases. The second set achieved an average rating of 2.95/4, with 75% of ratings at 3 or 4.

  • Fair comparisons and generalization (SahZ, 1xxz). We clarified the historical inputs used by different baselines, particularly the differences between STREAM configurations. We also investigated Qwen3-8B, LoRA fine-tuning, and loss-weight sensitivity. These experiments clarify our design choices and show that a newer or fine-tuned language model does not necessarily improve performance under the same training configuration. We further clarified that the 14 pathology prototypes provide auxiliary supervision rather than restricting the generated report vocabulary.

Closing remarks.

The rebuttal has substantially strengthened our work, particularly through the requested leave-one-out ablation, direct progression evaluation, matched BCE comparison, methodological refinement, and additional clinical assessments. We believe these new results address several central concerns regarding the effectiveness and soundness of our framework, while the remaining limitations have been explicitly acknowledged.

We sincerely hope the reviewers will reconsider their assessments and ratings in light of the additional experimental evidence and substantial revisions.

Thank you again for your careful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.