acceptodds
Withdrawn from ICLR 2027

SIGMA: Spatiotemporal Interventional Generation via Mechanism-Aware Unified Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Clinical management of blinding fundus diseases is a long-term process involving continuous follow-up and dynamic intervention. Beyond assessing the current pathological status, clinicians also need to evaluate how lesions may evolve over a future time horizon under a given treatment strategy. Existing medical vision AI is mainly organized around two task formulations: static perception, which focuses on the current pathological state in single-time-point images, and \medical image editing, which focuses on controlled image modification under given target attributes or editing constraints. These two paradigms address current-state analysis and target-appearance control, respectively, but neither explicitly models the relationship among treatment strategy, time horizon, and pathological evolution, making them unsuitable for future image forecasting under intervention constraints. To address this gap, we introduce Spatiotemporal Interventional Generation (SIG) Task. Given a baseline image, intervention variables, and a time horizon, SIG requires the model to jointly predict the future image and its structured clinical rationale. For this task, we propose SIGMA, a tool-augmented unified generation-understanding model. SIGMA first extracts discrete biomarker representations from the baseline image and uses them to filter an external memory, where candidates are ranked by image similarity. The retrieved trajectories, intervention, and time horizon then condition unified forecasting. Under such priors, the unified model jointly generates the clinical rationale and the future image. Experiments reveal the challenges that SIG poses to existing models and demonstrate the effectiveness of SIGMA after task-specific fine-tuning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Medical AI, Spatiotemporal Forecasting, Unified Multimodal Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vFTaOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Spatiotemporal Interventional Generation, a task for forecasting future medical image states under treatment conditions and temporal constraints. The authors propose SIGMA, a unified generation-understanding framework that combines biomarker-aware perception and trajectory retrieval to predict future OCT images with clinical rationales. They also construct the Ophtha-SIG benchmark for longitudinal intervention-aware forecasting. Experiments show strong performance in image prediction, rationale generation, and cross-modal consistency. Overall, the paper presents an interesting direction for longitudinal medical AI, although some concerns remain regarding causal interpretation and evaluation methodology.

Critical strengths

The main strength of this paper is the introduction of a new and clinically meaningful task formulation for longitudinal medical image forecasting. Moving beyond static diagnosis and image editing, the proposed SIG setting explicitly models future disease evolution under treatment conditions, which is an important step toward more proactive medical AI systems. Another significant strength is the construction of the Ophtha-SIG benchmark, which provides longitudinal OCT data with intervention information, biomarker transitions, and clinical rationales, enabling systematic evaluation of this emerging problem.

Critical weaknesses

A key weakness of this paper is that the term “interventional” may overstate the causal interpretation supported by the current experimental setting. Since the dataset is constructed from retrospective observational records, the model primarily learns treatment-conditioned future prediction rather than estimating counterfactual outcomes under explicit interventions. This distinction is important for the claimed clinical implications. In addition, the comparison with existing multimodal baselines is not fully conclusive because SIGMA is fine-tuned on the proposed benchmark while the extent of task-specific adaptation for competing methods is unclear. These issues make it difficult to determine whether the gains come from the proposed framework itself or from the additional supervision and benchmark specialization.

Questions for authors
  1. SIGMA is fine-tuned on the Ophtha-SIG benchmark, while several baseline models appear to be evaluated without equivalent task-specific adaptation. Could the authors provide results for fine-tuned versions of the strongest baselines under the same training protocol to better isolate the contribution of the proposed framework?
  2. The clinical rationales are generated using an LLM and filtered by biomarker consistency checks. Could the authors provide more details on the degree of expert involvement in rationale validation and discuss how they ensure that improvements in rationale metrics reflect genuine clinical reasoning rather than similarity to LLM-generated references?
Reviewer_1xueOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This is an application targeted paper regarding fundus treatment. A clear problem is the development of methods that can assess how fundus treatment evolves over time. Current methods either focus on how the disease state currently exists or attempts to focus on image modification under a given target attribute. The authors refer to this as medical image editing. In order to enable this assessment of treatment over time, the authors introduce a spatiotemporal intervention task (SIG). SIG requires prediciting of the future image and its clinical rationalization based on an image, intervention information, and knowledge of the time horizon. The authors curate a dataset in order to enable this task. The authors also build the SIGMA framework that is thought of a tool augmented unified generation understanding model that essentially acts as a training framework that leverages this dataset. The authors shows its improvement on a variety of tasks.

Critical strengths
  1. The authors present a dataset that clearly addresses a known gap in medical imaging datasets which is datasets that exhibit treatment trends during some type of intervention. They provide more detail than comparable datasets by having physicians provide a dynamic rationale for the reasoning for logitudinal changes. This type of dynamic rationale data is difficult to curate and is a real point towards building trajectory prediction image models.

  2. The authors task development is novel as well. The SIG task can serve as a future benchmark for many medical imaging tasks as its very unclear whether models can really identify fine-grained differences across time that are very relevant to the medical field. I find it interesting to understand how the dynamic rationale of the treatment helps with the future image prediction task.

  3. The authors show clear improvement of their method over previous medical imaging models.

Critical weaknesses
  1. I don't see any details about the dataset development process. I highlighted this as an ethics concern. Please read what I wrote in that section.

  2. There is a real novelty concern regarding the development of their SIGMA framework. Essentially, their framework has three steps: Use a Dino v3 background to identify biomarkers from the data, extract comparable samples from a memory back with Dino v3 embeddings, and then fine-tune on top of an already pre-trained MoT model that is initialized with weights from the previous BAGEL paper. Essentially, their framework is a processing step followed by fine-tuning with data that they were able to curate on top of a method that did most of the pre-training already. Essentially, the only real novelty in this whole process is showing how to use the data that they were able to curate that other methods did not have access to. Their method functions as an ablation study on top of the BAGEL paper, rather than something that teaches us how to develop a model that is functionally better for this new task. I think this paper could have been viewed as more novel if the benchmark was the star of the paper and the methology was more about how to tune any model towards this task and what framework helps take existing models towards this task. Instead, the novelty concerns begin because they are trying to frame fine-tuning on top of another model as a novel method and for that I have to assess what the authors provided.

  3. Since the authors are proposing a new method, it should be transferable to other datasets. Right now the whole paper centers on this one dataset where it isn't clear what the distributional and diversity characteristics are (since these are not shown in detail). The OLIVES dataset would serve as a good comparison. In the appendix, the authors state that the OLIVES dataset does not have dynamic rationale, but for the purposes of experimental ablation studies it is entirely possible to make rough estimates of their dynamics in order to enable a study. For example, they could use a rough VLM generation process like they did for this paper on the OLIVES dataset to get similar trajectory information that they could use to try and verify their method on another dataset. I understand that data is limited in this domain, but some creative attempt at trying to demonstrate generalizablility between datasets is needed. This is because there are so many medical datasets that cause an overfitting effect towards their own patient populations, but result in low transferability to other datasets.

  4. This setup becomes more problematic when they show the comparisons between their method and previous approaches. In almost all cases, the gap between their approach and previous approaches is massive because the only difference is that their approach was allowed to fine-tune on the data they provided while other methods were not allowed to perform fine-tuning. I do not consider this a fair comparison. The authors should have shown how these previous methods performed when some attempt at fine-tuning with their dataset is performed. I understand that certain models may not provide weights, but there are many open source models where some attempt at a fine-tuning setup should have been performed.

Questions for authors

Suggestions for Future: I notice that the abstract introduction is about fundus, but the images of the paper is about OCT. Was their any consideration to leverage the fundus photography images themselves as part of the multi-modal generation process.

Clarifications: How did you choose which biomarkers would be used in the study? The OLIVES dataset uses 16 biomarkers around a wide variety of information.

Which diseases are being targeted? Is it generally any disease or do the populations have AMD, DME, DR, etc. and there is something to try to understand specifically. I feel that some degreee of results stratification along different disease axes would help the benchmark novelty of the paper.

Reviewer_y2CQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Spatiotemporal Interventional Generation (SIG), a task that jointly predicts a future OCT image and a clinical rationale given a baseline scan, a treatment plan, and a time interval. The model also generates a text explanation of the predicted changes. The authors build the Ophtha-SIG dataset and propose SIGMA, which uses DINOv3 to identify biomarkers and uses a BAGEL model further trained on these data to generate the future image and its explanation. The experiments assess image quality, text quality and disease classification performance.

Critical strengths
  1. Predicting how medical images may change over time under different treatments is an important research topic. The model uses treatment and time information to predict these changes and generate future images.
  2. The model uses biomarkers from the baseline OCT and changes observed in similar patients to guide its predictions. This provides a useful way to connect image generation with clinical findings and previous cases.
  3. The paper also includes assessments by clinical experts and compares their scores to the automated evaluator.
Critical weaknesses
  1. Appendix A provides some dataset statistics, but the numbers of unique patients and eyes are not clearly reported. Information about the data sources is also missing and the meanings of sequences & stages need clearer definitions. It is also unclear how OCT images from different visits are selected and matched.
  2. Most evaluations focus on image quality, text quality, or agreement between the generated image and its explanation. However, an image can look realistic and agree with its explanation while still showing the wrong clinical change. Table 3 evaluates the model’s ability to diagnose the current disease state, but it does not show whether generated future images improve a downstream task. The authors should compare the predicted changes with actual findings at later visits, such as changes in biomarkers or treatment response to show the value of generating an image.
  3. SIGMA receives additional training on Ophtha-SIG, while most models in Table 2 are used without training for this task. It is therefore difficult to tell whether the gains come from the proposed tools or from the additional training. Table 4 includes “Base SIGMA,” but its training settings and relationship to BAGEL need to be explained more clearly. This baseline should also be included in the main comparison.
  4. In Table 4, Tool 1 alone achieves the best text scores, while Tool 2 alone achieves the best image scores. Combining them may provide a useful balance, but the paper does not clearly show why this balance is preferable. Table 4 also does not report CRVA or confidence intervals. Uncertainty estimates are needed to interpret the differences between variants.
  5. Using biomarkers and similar cases does not by itself show that the model has learned how a treatment causes changes in disease. Patients receiving different treatments may also differ in disease severity and previous treatment response. Therefore, showing different generated images for injection and observation is not enough to establish what would happen if the same patient received a different treatment. The authors should explain what they mean by learning disease mechanisms and limit their conclusions to what the experiments support.
  6. Earlier studies have already predicted OCT images after treatment from baseline scans, including Liu et al. [1] and Moon et al. [2]. ProgEmu [3] also generates medical images together with text explanations, although it uses a different type of input. The paper should explain more clearly how its task and method differ from these studies. Grouping previous work mainly into current disease assessment and image editing does not fully cover the most relevant research.

Reference:

  1. Liu, et al. "Prediction of OCT images of short-term response to anti-VEGF treatment for neovascular age-related macular degeneration using generative adversarial network." British Journal of Ophthalmology 104.12 (2020): 1735-1740.
  2. Moon, et al. "Prediction of anti-vascular endothelial growth factor agent-specific treatment outcomes in neovascular age-related macular degeneration using a generative adversarial network." Scientific reports 13.1 (2023): 5639.
  3. Ma, et al. "Towards interpretable counterfactual generation via multimodal autoregression." MICCAI 2025.
Questions for authors

See weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.