acceptodds
Under review as a conference paper at ICLR 2027

Modeling Continuous Spatio-Temporal Disease Evolution for Longitudinal Radiology Report Generation

Abstract

Radiology report generation aims to assist radiologists in drafting medical reports by automatically generating descriptive text from images, thereby reducing workload and improving diagnostic efficiency and accuracy. Existing report generation methods often treat disease changes under irregular temporal sampling as fixed-interval discrete jumps, resulting in inaccurate disease state characterization. To address it, we propose a novel **Co**ntinuous spatio-temporal **D**isease **E**volution framework for longitudinal **R**adiology **R**eport **G**eneration (CoDE-RRG), which models continuous disease evolution from arbitrary numbers of prior visits and multi-view images. Specifically, we propose a view-stable multi-view fusion module, where a primary view serves as an anchor to aggregate information from all auxiliary views. Departing from traditional discrete modeling of longitudinal changes, we first employ disease-guided dimensionality reduction to project high-dimensional visual features into a low-dimensional disease-specific state space, and then innovatively introduce Neural Ordinary Differential Equations to compute the continuous temporal evolution trajectories of these disease states. It allows for the construction of individualized disease progression tracks that align with actual physiological changes. Extensive experiments demonstrate that our method consistently improves the modeling of longitudinal disease evolution and current disease states, with relative gains of up to 2.3% in BLEU-4 and 4.6% in CheXbert F1 over state-of-the-art methods. Our code is available at https://anonymous.4open.science/r/CoDE-RRG-E007/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Radiology Report Generation, Spatio-temporal Modeling, Longitudinal Disease Evolution, Neural Ordinary Differential Equations

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_u1HhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoDE-RRG for longitudinal radiology report generation. It combines multiview fusion with a disease graph that evolves through neural ODEs over actual examination intervals and incorporates visual observations through recurrent updates. The resulting representations guide report generation. Experiments on MIMIC-CXR and Longitudinal-MIMIC evaluate report quality and component contributions.

Critical strengths
  1. Modeling disease representations continuously over actual examination intervals is a meaningful contribution to longitudinal RRG. Combining neural ODEs with disease graph interactions provides a coherent approach to irregularly sampled examinations, and the comparisons with RNN and RNN+T support its effectiveness.
Critical weaknesses
  1. A closely related comparison is missing. DELRER [1] also uses neural ODEs to model disease evolution and support longitudinal report generation. CoDE-RRG differs by modeling disease graphs along actual examination timelines, whereas DELRER models evolution along disease severity. However, the paper does not explain or evaluate the additional benefit relative to this existing use of ODEs in RRG. A comparison under a common evaluation protocol would help establish the contribution beyond the RNN and RNN+T baselines.

  2. The experiments do not isolate the contribution of the proposed multiview fusion design. In Table 3, removing AMF also restricts each examination to a single view, while Table 5 compares different numbers of input views. These experiments demonstrate the value of additional visual information but do not establish the benefit of using a primary view as the fusion anchor. Since AMF is presented as a separate contribution, comparisons with pooling and attention using learned queries, with identical input views, are needed to substantiate this design.

Reference:

[1] Wang et al. Disease Evolution-Informed Longitudinal Radiology Report Generation. CVPR Findings, 2026.

Questions for authors
  1. How does CoDE-RRG compare with DELRER under a common evaluation protocol, and what additional benefit comes from modeling actual examination intervals and disease interactions?

  2. Does AMF outperform simpler fusion methods when the available images and other model components are held fixed?

  3. Why do w/o PI in Table 3 and Visits = 1 in Table 13 produce different results when both are described as using only the current examination?

  4. How is the examination interval incorporated into the implemented ODE solver, given the inconsistent interval scaling in Equations 2 and 3 and Algorithm 1?

Reviewer_SW4POpenReview ↗
Rating
2(−1)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoDE-RRG, a disease evolution framework for longitudinal radiology report generation. It combines multi-view chest X-ray fusion, a graph-structured latent state evolved with a neural ODE, and prompt-based report generation. The proposed method is evaluated on MIMIC-CXR and its longitudinal subset, and shows improved performance over existing methods.

Critical strengths
  • The paper addresses a relevant problem in longitudinal radiology report generation.
  • The writing is clear and the overall structure of the paper is well organized.
  • The idea of using multi-view X-ray and irregular-time interval modeling is reasonable and well-motivated.
Critical weaknesses
  • The main contribution of the paper needs to be further clarified. For the multi-view X-ray fusion, it's reasonable to set the primary view as query and the other views as keys and values via cross-attention, which is straightforward and the authors should properly acknowledged existing works. Besides, its superiority over other adaptive fusion methods should be further justified. For the irregular-time interval modeling, similar ideas have been explored in existing works like Latent ODE (Rubanova et al., 2019). Disease evolution modeling has also been explored in radiology report generation. The authors should clarify the methodological advance in their approach compared to these existing methods.
  • In Table 9, the authors claim that CoDE-RRG demonstrates advantages in temporal consistency over PriorRG, but underperformance in GREEN Score and overall error rate. It weakens the potential of the proposed method in clinical practice, where clinical findings are the essential information for diagnosis. The authors are suggested to provide more analysis or discussion on why the proposed method produces more false findings, misidentification of anatomic locations, and errors.
  • The claim of long-intervals advantages is not well supported. In both Figure 4 and Table 12, CoDE-RRG does not show higher improvement over PriorRG in long-intervals. For example, in Table 12, CoDE-RRG has lower 14-F1 than PriorRG in both 90-365 days and >365 days. Besides, the benefits of a longer history is not consistently observed. In Table 13, two-visit shows higher performance than five-visit in the NLG metrics, which may suggest that more visits do not necessarily lead to better performance, or the model is not able to effectively utilize the information from more visits. The higher F1 in five-visit may be due to the higher recall with more false positives.
Questions for authors
  • Eq. (2) and (3) seem to be contradictory. Eq. (2) does not depend on \delta t, while Eq. (3) does. The scaling strategy in Eq. (3) and Algorithm 1 should be further clarified.
Reviewer_JfndOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CoDE, a framework for longitudinal radiology report generation using multi-view chest X-rays, irregular visit intervals, and clinical context. It combines attention-based view fusion with a Neural ODE and disease graph to evolve latent disease states, updating them with each new examination before conditioning report generation. Experiments show improvements over strong baselines on several language and clinical metrics. However, factual accuracy is mixed and the physiological validity of the learned trajectories is actually not verified in the paper.

Critical strengths
  1. The paper addresses a practical problem in longitudinal radiology report generation by considering both irregular visit intervals and varying image views.

  2. The designed framework is resonable to me. The disease-specific representations allow efficient temporal modeling, while the observation updates incorporate new imaging evidence at each visit.

  3. The experiments indicate promising results. The ablation studies, particularly the comparison using real, fixed, and shuffled timestamps, help demonstrate the value of temporal information.

Critical weaknesses
  1. My biggest concern is that the claim of modeling physiological disease evolution is not well validated. Supervision is available only at examination times, so the learned trajectories between visits may not reflect the actual disease progression.

  2. The advantage of Neural ODE over simpler temporal models needs further evidence. The improvements over the time-aware RNN are relatively modest, and no confidence intervals or results across multiple runs are provided in the paper.

  3. The clinical evaluation shows mixed results. Although CheXbert scores improve, the overall GREEN score is lower than PriorRG, with more false findings and localization errors, leading to the claim of improved clinical correctness quite marginal.

Questions for authors
  1. How do you validate that the learned trajectories reflect actual disease progression, rather than simply providing useful features for report generation?

  2. How are time intervals incorporated into the RNN+T baseline? Are its model capacity and training settings comparable to the proposed method, and are the improvements consistent across multiple runs?

Comments on the paper

1 comment

Dear Reviewers,

As the discussion deadline approaches, we would be very grateful for any further thoughts or feedback on our rebuttal. We sincerely appreciate the time and effort you have devoted to reviewing our work and would be happy to address any remaining concerns or questions.

Best regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.