- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents PAPE, a framework for longitudinal radiology report generation that combines progression alignment with pathology semantic learning. It models clinical change using the residuals between current and prior image and report representations, and aligns their progression directions while maintaining image–report consistency at each examination. The method also learns pathology prototypes from clinical observation labels through an inverse optimal transport objective, providing explicit semantic supervision for the visual features. These representations are fused with prior image–report context and used to condition a frozen language model for report generation. Experiments on Longitudinal-MIMIC show competitive performance on both language-generation and clinical efficacy metrics.
Critical strengths
The paper gives a clear formulation of longitudinal change by aligning the progression of image and report representations across examinations. This fits the task well and provides a direct way to model temporal clinical information.
Critical weaknesses
- In direction-alignment loss, all non-matching pairs in a batch are used as negatives. Patients can share similar clinical changes, so clinically related progression patterns may be pushed apart.
- Progression is defined by the difference between global image representations. This residual can also reflect changes in view, positioning, or acquisition, which are not separated from true clinical change.
- For studies with multiple images, features are averaged at corresponding token positions across views. Frontal and lateral radiographs have no direct spatial correspondence, making this averaging operation questionable before temporal modeling.
Questions for authors
- Does direction-alignment loss consider cases where different patients have similar clinical progression?
- Can view, positioning, or acquisition differences also affect the residual direction?
4 comments
Supplementary response to Reviewer GFs6
For Q1. Similar clinical progression
Thank you for raising this point. We acknowledge that instance-level contrastive objectives can introduce false negatives: unmatched image–report pairs may share clinical findings, and different longitudinal examination pairs may exhibit similar progression patterns. Our direction alignment aims to capture finding-specific clinical changes, rather than only broad categories such as “improving” or “worsening.” For example, improvement in pulmonary edema and improvement in pleural effusion involve different findings and need not represent the same semantic transition. Nevertheless, genuinely similar changes can occur across patients, and treating these pairs as negatives may penalize clinically meaningful similarities.
Alongside instance-level alignment, the label-guided prototype objective provides shared clinical semantic targets, while the report-generation objective supplies supervision from the full reports. These objectives provide complementary semantic constraints, although they do not explicitly eliminate false negatives. Our ablation results show that removing reduces both report-generation quality and clinical F1, supporting an overall benefit of progression alignment under the evaluated setting despite this limitation.
In future work, we plan to explore a semantics-aware soft alignment mechanism that adjusts the contribution of negative pairs according to their clinical similarity. For direction alignment, this could use label patterns across both prior and current examinations to identify potentially similar progression, assigning such pairs a reduced negative penalty. We have clarified this limitation and discuss this extension in the revised Appendix.
Thanks for the explanations and updated results. The revised loss helps, but I still have concerns about false negatives and averaging across views. I’m keeping my original score.
Further clarification to Review GFs6
Thank you for considering our explanations and updated results. We appreciate your remaining concerns and would like to clarify them as follows:
For false negatives
We acknowledge that our current formulation does not explicitly correct false negatives. Our rationale for retaining it is that instance-level cross-modal contrastive learning provides useful supervision from observed correspondences, even when some unmatched samples may share semantic content. This training principle continues to be used in recent work, including but not limited to TIM [1], Sparse CLIP [2], CLIP-FMoE [3], LIVE [4], and TILA [5]. Although their objectives and sampling strategies differ, their published formulations retain unmatched cross-instance pairs as negatives without explicitly identifying or downweighting semantically equivalent cross-instance pairs.
In our task, the ablation results provide direct evidence of the overall benefit: removing the direction-alignment loss reduces BLEU-4 from 0.136 to 0.128 and clinical F1 from 0.512 to 0.480. These results support retaining progression alignment in the current framework, while leaving room for improvement in how negatives are handled. We therefore acknowledge false negatives as an unresolved limitation and plan to investigate a semantics-aware soft alignment strategy that adjusts negative-pair weights according to clinical label similarity across both prior and current examinations.
[1] TIM: Temporal Decoupling with Iterative Mutual-Refinement Model for Longitudinal Radiology Report Generation. CVPR, 2026.
[2] Sparse CLIP: Co-Optimizing Interpretability and Performance in Contrastive Learning. ICLR, 2026.
[3] CLIP-FMoE: Scalable CLIP via Fused Mixture-of-Experts with Enforced Specialization. ICLR, 2026.
[4] Language-Instructed Vision Embeddings for Controllable and Generalizable Perception. ICLR, 2026.
[5] Temporal Inversion for Learning Interval Change in Chest X-Rays. CVPR, 2026.
For multi-view averaging
Our experiments use only frontal chest radiographs, including anteroposterior (AP) and posteroanterior (PA) projections; lateral radiographs are excluded. This restriction was intended to reduce representation differences arising from mixing frontal and lateral projections, so that longitudinal residuals could focus more closely on clinical changes within frontal radiographs. Our earlier response did not make this restriction sufficiently clear. When multiple eligible images are available for an examination, averaging is performed only among these frontal images. The resulting global token supports the auxiliary alignment objectives, while the full aggregated sequence is used for report generation.
Thus, the specific concern about averaging frontal and lateral anatomical layouts does not apply to our experiments. Nevertheless, we agree that differences in AP/PA projection and positioning can still affect spatial correspondence, and averaging remains a simple aggregation strategy. We have clarified the frontal-only input setting and its motivation in the implementation details and revise the limitations accordingly.
For both prior and current examinations, we use only frontal chest radiographs acquired in anteroposterior (AP) or posteroanterior (PA) projections and lateral radiographs are excluded. This restriction aims to reduce representation differences arising from mixing frontal and lateral projections, allowing longitudinal residuals to focus more closely on clinical changes within frontal radiographs. When an examination contains multiple eligible frontal images, their adapted token sequences are averaged at the same sequence indices to obtain an examination-level representation.
Thanks for you feedback again! If you have any further concerns, we would be happy to provide clarification and improve the manuscript based on your feedback.
Response to Reviewer GFs6
Thank you for your careful reading of the direction loss and the aggregation across views. In response, we revised the manuscript to explain the role of textual residuals in training and to discuss the limitations of contrastive alignment and view averaging. These changes make the method easier to follow and point to concrete ways to improve it.
For W1. Similar changes across patients
Thank you for raising this point. We acknowledge that instance-level contrastive objectives can introduce false negatives: unmatched image–report pairs may share clinical findings, and different longitudinal examination pairs may exhibit similar progression patterns. Our direction alignment aims to capture finding-specific clinical changes, rather than only broad categories such as “improving” or “worsening.” For example, improvement in pulmonary edema and improvement in pleural effusion involve different findings and need not represent the same semantic transition. Nevertheless, genuinely similar changes can occur across patients, and treating these pairs as negatives may penalize clinically meaningful similarities.
Alongside instance-level alignment, the label-guided prototype objective provides shared clinical semantic targets, while the report-generation objective supplies supervision from the full reports. These objectives provide complementary semantic constraints, although they do not explicitly eliminate false negatives. Our ablation results show that removing Ldir reduces both report-generation quality and clinical F1, supporting an overall benefit of progression alignment under the evaluated setting despite this limitation.
In future work, we plan to explore a semantics-aware soft alignment mechanism that adjusts the contribution of negative pairs according to their clinical similarity. For direction alignment, this could use label patterns across both prior and current examinations to identify potentially similar progression, assigning such pairs a reduced negative penalty. We will clarify this limitation and discuss this extension in the revised manuscript.
For W2 and Q2. Acquisition effects on residuals
Thank you for pointing this out. We agree that differences between global image representations can reflect changes in projection, positioning, or acquisition conditions, in addition to clinical progression. Following your suggestion, we revised the direction-alignment objective to use textual residuals as detached semantic references for learning visual residuals:
Ldir=C(Dv,sg[Dt]).
We retain both contrastive terms and stop gradients through the textual residuals in both terms. Consequently, Ldir updates the visual adapter, while the text adapter continues to receive supervision from the other training objectives. This revision encourages visual residuals to align with the clinical changes described in the reports, using the textual residuals as semantic guidance.
With the revised formulation, BLEU-4 increases from 0.134 to 0.136, METEOR from 0.171 to 0.174, and clinical F1 from 0.510 to 0.512. These results show improved report-generation quality and clinical agreement with the revised model. We have updated the methodological description and experimental results accordingly.
For W3. Aggregation across views
Thank you for pointing this out. We agree that frontal and lateral radiographs do not have direct anatomical correspondence at matching token positions. Our current averaging operation is a simple aggregation heuristic for obtaining a fixed-length examination representation; it does not establish spatial correspondence between views.
We would also like to clarify that the progression-direction loss operates on examination-level global representations obtained by averaging the global tokens from individual images. It therefore does not compute temporal differences between spatially matched patch tokens. However, the full token sequences used for report generation are also averaged across views before semantic compression and temporal fusion. We acknowledge that this operation can mix features from different anatomical regions and lose view-specific information.
We have already clarified the limitations in the appendix. A natural improvement is to preserve the token sequence from each view and let the Perceiver attend jointly to the concatenated sequences, avoiding position-wise averaging before temporal modeling. We plan to investigate this view-preserving aggregation strategy in future work.