Silent Failures in Neuroimaging Generative Models: Falsification-Oriented Evaluations

Generative models in neuroimaging are typically evaluated with structural- and intensity-based metrics (SSIM, MSE, PSNR, FID/KID) inherited from natural-image generation and from classical registration, reconstruction, and denoising algorithms whose limited expressive power made such scores a reasonably safe proxy for correctness. We show that this convention becomes actively misleading for deep generative models in neuroimaging, where the scientifically or clinically relevant signal is typically a small deviation from a highly stable, shared anatomical pattern. We replicated two published, openly available models to serve as case studies: BrLP, a latent-diffusion disease progression model, and NT-ViT, an EEG-conditioned fMRI generator, and found that both achieve image-level scores matching their original publications, yet fail to falsify basic claims about task performance once evaluated against trivial baselines (an identity mapping, random subject matching) and task-specific metrics: BrLP's volumetric and directional predictions do not outperform a naive heuristic, and NT-ViT's reconstructions show no measurable relationship to the ground-truth time series despite favorable SSIM and PSNR. Controlled simulations confirm the mechanism: SSIM and MSE remain near ceiling even when all temporal information in an fMRI run is removed, because both are dominated by static, shared anatomy rather than the dynamics of interest. Using a lightweight fMRI tokenizer, we further show that explicit inductive biases in data preprocessing and the training objective expose fidelity/task-performance tradeoffs invisible to any single metric. We argue for evaluation grounded in task-relevant, falsification-oriented metrics, contextualized by task-appropriate baselines, rather than reliance on standard structural and intensity-based scores.