Making Longitudinal Context Matter for Radiology Report Generation

Longitudinal radiology report generation requires interpreting historical evidence in the context of the current examination. However, static historical representations lack explicit patient-specific adaptation, and simply combining prior images and reports with the current image does not establish their effective contribution. Moreover, full-context cross-entropy supervision can reduce generation loss despite neglecting historical visual evidence or relying primarily on textual patterns in prior reports. We propose PHAM-CECA, an end-to-end framework that couples patient-specific historical representation with contribution-aware optimization. Patient-specific Historical Associative Memory (PHAM) writes prior visual and textual evidence into separate fast-weight memories through patient-specific adaptation. The current image queries both memories to retrieve historical representations tailored to the present examination. The proposed Calibrated Expected Context Advantage (CECA) measures visual, textual, and overall historical contributions by evaluating the same target report under full-history, text-history, image-history, and current-only conditions. PHAM adapts memory parameters per case in the inner loop, while generation supervision and CECA optimize shared parameters in the outer loop. Experiments on the Longitudinal-MIMIC dataset demonstrate the effectiveness of the proposed method.