- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper investigates whether standard CV metrics adequately evaluate generative models designed for neuroimaging tasks. It presents two cases, BrLP - for longitudinal MRI generation, and NTViT for EEG to fMRI generation. The findings are that conventional image similarity metrics can be dominated by shared anatomical structure and therefore obscure failures to preserve task relevant info. Additional controlled simulations and fMRI tokeniser experiments demonstrate discrepancies between structural fidelity, temporal info, and downstream task performance.
Critical strengths
The paper addresses a relevant research questions relating to the evaluation of generative models for neuroimaging, with structured and controlled experimentation. Inclusion of simple baselines allow for interpretable results and analysis.
The controlled fMRI experiments demonstrate how favourable image similarity metrics can coexist with loss of temporal information, and the released code, experimental setups, and results supported independent verification.
Critical weaknesses
-
The longitudinal disease-progression evaluation does not sufficiently characterise how BrLP provided predictive value: The authors advocate for task-relevant evaluation, but several aspects of the BrLP analysis do not fully establish whether the model captures individualised disease progression. First - volumetric prediction errors are primarily aggregated across follow up horizons. This is potentially misleading because an identity baselines is naturally competitive at short intervals, when anatomical changes will be small. (The authors report that BrLP becomes more competitive at longer horizons. ) The more informative comparison is therefore whether BrLP improves upon identity and simple volumetric predictors as the magnitude of true anatomical change increases. Although fig 1 stratifies MSE and SSIM by follow up interval, the corresponding analysis is missing volumetric MAE, which is more directly relevant to the task. Second, the interpretation of directional accuracy requires additional care. Hippocampal atrophy and ventricular expansion are established trends in ageing and Alzheimer's disease. Consequently, a naive heuristic predicting these directions can achieve high accuracy without modelling individualised progression. Conversely, small anatomical changes (over short intervals) can have their estimated direction reverse by relatively minor reconstruction/segmentation errors. Directional accuracy is therefore useful as a supplementary measure but should not substitute evaluating the accuracy of predicted volumetric changes. Finally, the authors characterise BrLPs predicted CSF as approx constant over time, citing low R^2 values and slopes of +0.6%/year (OASIS-2) and +0.41%/year (ADNI). However, low pooled R^2 does not establish weak within-patient responsiveness to time, particularly in the presence of substantial between-patient heterogeneity. Nor are the reported slopes necessarily negligible without comparison to observed progression rates. A longitudinal analysis accounting for repeated observations would better support this conclusion.
-
fMRI evaluations do not fully isolate the claimed failure mechanisms: The NTViT evaluation reports near zero Pearson correlation and negative R^2 and explained variance, providing evidence of poor agreement with observed fMRI time series. However, the large discrepancy between R^2 and explained variance suggests that systematic mean offsets may contribute substantially to the former. These errors should be distinguished from failures to capture temporal dynamics. Importantly, the evaluation does not establish if NTViT is meaningfully using its EEG conditioning. Poor correspondence with GT time series could be as a result of poor exploitation of EEG information or inaccurate temporal predictions. Clarifying how EEG–fMRI alignment, hemodynamic lag, and dataset-specific TRs were handled would help distinguish model failure from problems in the evaluation pipeline. Finally, several reporting inconsistencies require correction. Sec 3.1 and Discussion attribute approx 0.96 SSIM and 0.002 MSE to BrLP, whereas Table 1 assigns these values to identity. Section 3.2.3 reports LCFV ≈ −0.5, while Table 2 reports approximately −0.05.
Questions for authors
1 ) Longitudinal progression: Could the authors extend the horizon-stratified analysis in Figure 1 to volumetric MAE for all five evaluated regions, covering the full ADNI follow-up range? Additionally, could they compare predicted and observed volumetric progression as a function of follow-up time, accounting for repeated measurements and patient-level heterogeneity, with uncertainty estimates? This would help establish whether BrLP provides additional predictive value at longer horizons and whether its predicted progression rates are meaningfully responsive to time.
- EEG conditioning and temporal fidelity: Could the authors compare NTViT predictions using correctly paired EEG, temporally shifted EEG, and EEG from another subject? Please clarify how EEG–fMRI temporal alignment and hemodynamic lag were handled, and report subject-level temporal correlations alongside comparisons against a constant mean-volume baseline. These controls would help determine whether the observed failures arise from insufficient EEG-dependent information or from other reconstruction and evaluation errors.