acceptodds
Under review as a conference paper at ICLR 2027

Making Longitudinal Context Matter for Radiology Report Generation

Abstract

Longitudinal radiology report generation requires interpreting historical evidence in the context of the current examination. However, static historical representations lack explicit patient-specific adaptation, and simply combining prior images and reports with the current image does not establish their effective contribution. Moreover, full-context cross-entropy supervision can reduce generation loss despite neglecting historical visual evidence or relying primarily on textual patterns in prior reports. We propose PHAM-CECA, an end-to-end framework that couples patient-specific historical representation with contribution-aware optimization. Patient-specific Historical Associative Memory (PHAM) writes prior visual and textual evidence into separate fast-weight memories through patient-specific adaptation. The current image queries both memories to retrieve historical representations tailored to the present examination. The proposed Calibrated Expected Context Advantage (CECA) measures visual, textual, and overall historical contributions by evaluating the same target report under full-history, text-history, image-history, and current-only conditions. PHAM adapts memory parameters per case in the inner loop, while generation supervision and CECA optimize shared parameters in the outer loop. Experiments on the Longitudinal-MIMIC dataset demonstrate the effectiveness of the proposed method.

Primary area:
generative models
Keywords:
Longitudinal radiology report generation, Historical associative memory, Patient-specific adaptation, Contribution-aware optimization

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_LTz1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PHAM-CECA to make prior images and reports more useful for longitudinal radiology report generation. PHAM uses separate patient-adapted fast-weight memories for historical visual and textual evidence, queried by the current image. CECA compares the same target report under full-history, text-history, image-history, and current-only contexts, and combines likelihood-advantage constraints with multi-context generation supervision. The paper also analyzes the induced retrieval mechanism and the context-advantage objective. On Longitudinal-MIMIC, PHAM-CECA achieves clinical F1 of 0.483 and ROUGE-L of 0.296. While the approach is well motivated, the evaluation does not directly establish better temporal-change reasoning, and the improvement over the full-history baseline is modest (F1: 0.477 → 0.483).

Critical strengths
  1. Task-relevant memory design. PHAM adapts separate image and report memories to each patient and uses the current image to retrieve historical evidence, rather than relying on static history fusion.

  2. Well-motivated optimization. CECA compares the same target across four contexts and retains reduced-context supervision to avoid directly rewarding degraded ablated-context predictions. The theoretical analysis makes the objective's assumptions and limitations clearer.

  3. Useful ablations. PHAM improves clinical F1 over Static (0.465 vs. 0.429, Table 2); combining reduced-context supervision and advantage constraints achieves F1 0.483, compared with 0.474 and 0.469 when either component is used alone (Table 4).

Critical weaknesses
  1. Temporal reasoning is not directly evaluated. CheXbert F1 measures clinical findings but does not verify whether a finding is new, persistent, improved, or resolved. The t-SNE plots and selected report examples are informative, but do not establish temporal-change accuracy over the test set. This leaves the paper's central longitudinal claim insufficiently tested.

  2. Small gains without sufficient robustness evidence. Clinical F1 improves only from 0.477 (full-history baseline) to 0.483. The paper reports confidence intervals for context-advantage analyses, but not paired intervals or multi-seed variability for the main clinical gains. PHAM's benefit over a parameter-matched static, current-conditioned retriever is also unclear, limiting attribution to gradient-based adaptation.

Questions for authors
  1. Report quantitative results for new, persistent, improved, and resolved findings; can the model distinguish cases with identical current labels but different historical states?

  2. Provide paired confidence intervals or significance tests for the main F1 gains, accounting for multiple studies per patient where appropriate; report variability across training seeds.

  3. Clarify the key/query and memory input/output dimensions relative to the stated 768→2048→4096 MLP, and specify how CECA's lower confidence bounds and calibration split are defined.

Reviewer_aqMeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes PHAM-CECA for longitudinal radiology report generation, using patient-specific visual and textual memories queried by the current image. Its contribution-aware training objective compares report likelihoods across four context settings to encourage effective use of historical information. Theoretical analysis and experiments on Longitudinal-MIMIC, including baseline comparisons, ablations, and qualitative studies, support the approach and show improvements in textual similarity and clinical-label agreement.

Critical strengths
  1. Separate adaptive memories for prior images and reports enable historical retrieval tailored to the current examination.
  2. Baseline comparisons, component ablations, and retrieval diagnostics provide evidence for the benefits of both proposed modules.
Critical weaknesses
  1. The dimensions are inconsistent across Eqs. (1)–(3), Appendix A, and the implementation. In particular, Appendix A assumes equal memory input and output dimensions, while the reported architecture is 768→2048→4096. Please clarify the actual dimensions.

  2. The evaluation does not directly assess temporal accuracy. CheXbert labels cannot distinguish a new finding from a persistent one when the current disease labels are identical. A direct evaluation of interval changes would better support the main claim.

  3. The F1 gain over the full-history baseline is modest: 0.477 to 0.483. Without repeated runs or confidence intervals, the reliability of this improvement is unclear.

  4. The 2,058 test samples come from 266 patients. The statistical analysis should account for correlations between examinations from the same patient, for example through patient-level resampling.

  5. The discussion of Table 4 states that advantage constraints alone lower F1, yet F1 increases from 0.466 to 0.469 relative to full-context supervision. Please clarify the comparison or correct the statement.

Questions for authors
  1. What projection and memory dimensions were actually used in the experiments? How do these reconcile Eqs. (1)–(3), Appendix A, and the reported 768→2048→4096 architecture?

  2. Is the F1 improvement from 0.477 to 0.483 consistent across runs? What uncertainty estimates are available, and do they account for repeated examinations from the same patient?

  3. What quantitative evidence supports improved accuracy in describing interval changes, such as distinguishing new from persistent findings, beyond the reported gains in current disease-label F1?

Reviewer_9nL3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In longitudinal radiology report generation, the paper focuses on the lack of explicit patient-specific adaptation in historical representations and how to learn to make historical information genuinely improve predictions. It proposes the Patient-specific Historical Associative Memory (PHAM) and Calibrated Expected Context Advantage (CECA) modules. The former is responsible for constructing patient-specific historical representations, while the latter encourages this historical information to contribute to predictions through training objectives. The theoretical analysis of the adaptation and optimization mechanisms, together with experiments on Longitudinal-MIMIC, supports the effectiveness of the framework.

Critical strengths

This paper distinguishes between providing patient-specific historical information and learning how to benefit from historical information in its motivation, yet combines them during the training process, presenting a coherent architecture that goes beyond simply aligning and understanding historical inputs. The ablation analysis, adaptation diagnostics, and intervention experiments on memory queries and historical associations in the paper are helpful in supporting its claims.

Critical weaknesses
  1. Compared to the full-history baseline, the improvement of the proposed method is relatively limited (F1 from 0.477 to 0.483, while ROUGE-L from 0.293 to 0.296), and reporting the standard deviation across multiple independent training runs might be beneficial.

  2. Table 4 shows that the proposed method achieves higher scores in Clinical Efficacy and Context Advantage. However, as they mentioned in their introduction, the increased gap between full and reduced contexts does not necessarily mean better utilization of history. This table does not report the changes in each context score separately, making it unclear where the increased advantage comes from. Reporting these changes would be helpful.

Questions for authors
  1. Can you clarify which parameters are updated during inference, and whether the shared initialization remains fixed? For each case, does the historical reconstruction update step start from the learned shared memory initialization?
  2. Are the backbone network, trainable parameters, training phase, and training budget of the full-history baseline in Table 1 consistent with those of the main model?
Reviewer_QsyFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates longitudinal radiology report generation and proposes PHAM-CECA to improve the utilization of historical images and reports. The method combines patient-specific historical memory with a context-aware optimization strategy to enhance the integration of longitudinal information. Experiments on Longitudinal-MIMIC demonstrate improvements over existing approaches. Overall, the paper addresses a meaningful problem and presents an interesting framework. However,

Critical strengths
  1. The paper addresses an important and relatively challenging problem in longitudinal radiology report generation, namely how to effectively utilize historical images and reports rather than simply incorporating them as additional inputs. This problem is practically meaningful and relevant to clinical applications.

  2. The proposed PHAM-CECA framework provides a reasonable approach to this problem by combining patient-specific historical memory adaptation with contribution-aware optimization. In particular, evaluating historical information under four different input contexts offers an explicit way to investigate its contribution to report generation. The theoretical analysis and ablation studies further support the design of the proposed framework.

Critical weaknesses
  1. The performance gains over the full-history baseline are relatively modest, raising concerns about the actual benefits of the proposed framework. Additionally, given the additional complexity introduced by patient-specific memory adaptation and four-context optimization, the authors should provide a more thorough analysis of computational overhead and demonstrate that the performance improvements justify the additional complexity.
  2. The CECA objective optimizes context advantages at the batch level, encouraging positive average contributions from historical information. However, this does not necessarily guarantee that historical information benefits individual patients consistently. In longitudinal radiology, the relevance of prior examinations can vary substantially across patients. The current formulation may therefore overlook cases where historical information provides misleading or even detrimental signals. More fine-grained analysis of patient-level context contributions would strengthen the claims regarding the effectiveness of CECA.
  3. The evaluation does not sufficiently establish the model's ability to perform reliable temporal reasoning. Although the paper reports improvements in conventional text generation and clinical efficacy metrics, these metrics do not directly measure the accuracy of identifying new, persistent, resolved, or worsening findings across examinations. Since temporal reasoning is the central motivation of this work, more targeted temporal evaluation and clinical assessment are needed to demonstrate that the proposed framework genuinely improves longitudinal understanding rather than primarily improving report generation metrics.
Questions for authors
  1. Given the modest performance gains, could the authors quantify the additional computational costs of PHAM and CECA to justify the increased complexity?
  2. Could the authors provide patient-level evidence that improved context advantage leads to better clinical report quality, and quantitative results demonstrating more accurate identification of new, persistent, and resolved findings?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.