acceptodds
Under review as a conference paper at ICLR 2027

Silent Failures in Neuroimaging Generative Models: Falsification-Oriented Evaluations

Abstract

Generative models in neuroimaging are typically evaluated with structural- and intensity-based metrics (SSIM, MSE, PSNR, FID/KID) inherited from natural-image generation and from classical registration, reconstruction, and denoising algorithms whose limited expressive power made such scores a reasonably safe proxy for correctness. We show that this convention becomes actively misleading for deep generative models in neuroimaging, where the scientifically or clinically relevant signal is typically a small deviation from a highly stable, shared anatomical pattern. We replicated two published, openly available models to serve as case studies: BrLP, a latent-diffusion disease progression model, and NT-ViT, an EEG-conditioned fMRI generator, and found that both achieve image-level scores matching their original publications, yet fail to falsify basic claims about task performance once evaluated against trivial baselines (an identity mapping, random subject matching) and task-specific metrics: BrLP's volumetric and directional predictions do not outperform a naive heuristic, and NT-ViT's reconstructions show no measurable relationship to the ground-truth time series despite favorable SSIM and PSNR. Controlled simulations confirm the mechanism: SSIM and MSE remain near ceiling even when all temporal information in an fMRI run is removed, because both are dominated by static, shared anatomy rather than the dynamics of interest. Using a lightweight fMRI tokenizer, we further show that explicit inductive biases in data preprocessing and the training objective expose fidelity/task-performance tradeoffs invisible to any single metric. We argue for evaluation grounded in task-relevant, falsification-oriented metrics, contextualized by task-appropriate baselines, rather than reliance on standard structural and intensity-based scores.

Primary area:
applications to neuroscience & cognitive science
Keywords:
generative models, disease models, MRI, neuroimaging, diffusion, tokenizer, autoencoder, SSIM, metrics, falsification

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_3FeoOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates whether standard CV metrics adequately evaluate generative models designed for neuroimaging tasks. It presents two cases, BrLP - for longitudinal MRI generation, and NTViT for EEG to fMRI generation. The findings are that conventional image similarity metrics can be dominated by shared anatomical structure and therefore obscure failures to preserve task relevant info. Additional controlled simulations and fMRI tokeniser experiments demonstrate discrepancies between structural fidelity, temporal info, and downstream task performance.

Critical strengths

The paper addresses a relevant research questions relating to the evaluation of generative models for neuroimaging, with structured and controlled experimentation. Inclusion of simple baselines allow for interpretable results and analysis.

The controlled fMRI experiments demonstrate how favourable image similarity metrics can coexist with loss of temporal information, and the released code, experimental setups, and results supported independent verification.

Critical weaknesses
  1. The longitudinal disease-progression evaluation does not sufficiently characterise how BrLP provided predictive value: The authors advocate for task-relevant evaluation, but several aspects of the BrLP analysis do not fully establish whether the model captures individualised disease progression. First - volumetric prediction errors are primarily aggregated across follow up horizons. This is potentially misleading because an identity baselines is naturally competitive at short intervals, when anatomical changes will be small. (The authors report that BrLP becomes more competitive at longer horizons. ) The more informative comparison is therefore whether BrLP improves upon identity and simple volumetric predictors as the magnitude of true anatomical change increases. Although fig 1 stratifies MSE and SSIM by follow up interval, the corresponding analysis is missing volumetric MAE, which is more directly relevant to the task. Second, the interpretation of directional accuracy requires additional care. Hippocampal atrophy and ventricular expansion are established trends in ageing and Alzheimer's disease. Consequently, a naive heuristic predicting these directions can achieve high accuracy without modelling individualised progression. Conversely, small anatomical changes (over short intervals) can have their estimated direction reverse by relatively minor reconstruction/segmentation errors. Directional accuracy is therefore useful as a supplementary measure but should not substitute evaluating the accuracy of predicted volumetric changes. Finally, the authors characterise BrLPs predicted CSF as approx constant over time, citing low R^2 values and slopes of +0.6%/year (OASIS-2) and +0.41%/year (ADNI). However, low pooled R^2 does not establish weak within-patient responsiveness to time, particularly in the presence of substantial between-patient heterogeneity. Nor are the reported slopes necessarily negligible without comparison to observed progression rates. A longitudinal analysis accounting for repeated observations would better support this conclusion.

  2. fMRI evaluations do not fully isolate the claimed failure mechanisms: The NTViT evaluation reports near zero Pearson correlation and negative R^2 and explained variance, providing evidence of poor agreement with observed fMRI time series. However, the large discrepancy between R^2 and explained variance suggests that systematic mean offsets may contribute substantially to the former. These errors should be distinguished from failures to capture temporal dynamics. Importantly, the evaluation does not establish if NTViT is meaningfully using its EEG conditioning. Poor correspondence with GT time series could be as a result of poor exploitation of EEG information or inaccurate temporal predictions. Clarifying how EEG–fMRI alignment, hemodynamic lag, and dataset-specific TRs were handled would help distinguish model failure from problems in the evaluation pipeline. Finally, several reporting inconsistencies require correction. Sec 3.1 and Discussion attribute approx 0.96 SSIM and 0.002 MSE to BrLP, whereas Table 1 assigns these values to identity. Section 3.2.3 reports LCFV ≈ −0.5, while Table 2 reports approximately −0.05.

Questions for authors

1 ) Longitudinal progression: Could the authors extend the horizon-stratified analysis in Figure 1 to volumetric MAE for all five evaluated regions, covering the full ADNI follow-up range? Additionally, could they compare predicted and observed volumetric progression as a function of follow-up time, accounting for repeated measurements and patient-level heterogeneity, with uncertainty estimates? This would help establish whether BrLP provides additional predictive value at longer horizons and whether its predicted progression rates are meaningfully responsive to time.

  1. EEG conditioning and temporal fidelity: Could the authors compare NTViT predictions using correctly paired EEG, temporally shifted EEG, and EEG from another subject? Please clarify how EEG–fMRI temporal alignment and hemodynamic lag were handled, and report subject-level temporal correlations alongside comparisons against a constant mean-volume baseline. These controls would help determine whether the observed failures arise from insufficient EEG-dependent information or from other reconstruction and evaluation errors.
Reviewer_6rGFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that image-level metrics (SSIM, MSE, PSNR, and to a lesser degree FID/KID) cannot falsify task claims for generative models in neuroimaging. The reason given is that most image variance comes from shared anatomy, while the signal of interest is a small deviation from it. The evidence has three parts:

  1. BrLP is a latent-diffusion disease progression model. The authors re-evaluate it on ADNI and OASIS-2 with the released weights. They compare it against an identity baseline (copy the baseline scan), two random matched-subject baselines, an XGBoost regressor on volumes, and a “naive” direction rule (grey matter shrinks, CSF and ventricles grow). The identity baseline beats BrLP on SSIM and MSE. BrLP’s ROI volume errors are higher than the baselines. On directional accuracy, BrLP does not clearly beat the naive rule. It ties on CSF by predicting an increase in essentially every case, by a roughly constant 6-8%.

  2. Replacing an fMRI run with its temporal mean plus noise leaves SSIM near ceiling and MSE barely changed. ROI Pearson, R2R^2, explained variance and a Log-Euclidean distance between ROI covariance matrices do detect the loss.

  3. The NT-ViT (EEG-to-fMRI) replication reproduces the published SSIM/RMSE/PSNR, but ROI time courses show ∣ρ∣≤0.021|\rho| \le 0.021 and R2≤−78R^2 \le -78.

A small BSQ fMRI tokenizer study is added to show that the scaling strategy and a correlation-based loss (SPC) trade off reconstruction fidelity against task metrics. The paper closes with three recommendations: task-specific metrics, trivial baselines, and inductive biases at design time.

Critical strengths
  1. The NT-ViT result is concrete and worth having in the literature. The authors reproduce the published SSIM/RMSE/PSNR of a peer-reviewed EEG-to-fMRI model to within a small margin. They then show that the ROI time courses of its outputs are essentially uncorrelated with the ground truth (∣ρ∣≤0.021|\rho| \le 0.021 across both datasets and both atlases). If this holds up, a published conditional model carries no recoverable information about the BOLD time course, and the metrics it was published with could not have shown that. This is a falsifiable, specific finding rather than a general complaint about metrics. It is the main reason I see value in this submission.

  2. The baselines are cheap, reproducible, and easy for others to adopt. Both case studies use the original code, and BrLP also uses the original weights. The evaluation code is released. The baselines (identity, random matched subject, temporal mean plus noise) can be added to almost any evaluation at no cost. The identity scan beating BrLP on the very metrics BrLP was published with is an easy-to-replicate data point. The disease-progression community should see it.

Critical weaknesses
  1. The BrLP case study does not isolate what the paper attributes to it.

(a) The metric claim and the baseline claim are run together:

  • The headline thesis is that structural and intensity metrics are blind to task failure. For BrLP, Table 1 does not show that.
  • SSIM and MSE rank the identity baseline above BrLP on both datasets. On OASIS-2 the figures are 0.962 vs 0.930 SSIM and 0.0021 vs 0.0050 MSE; ADNI is similar.
  • So once a no-change baseline is present, the conventional metrics catch the problem. The BrLP experiment supports “evaluate against persistence baselines”, which is already standard in longitudinal forecasting (e.g., the last-visit benchmark in the TADPOLE challenge). It does not support “SSIM/MSE are silent”.
  • Relatedly, the Discussion (l. 414) gives BrLP’s image-level scores as SSIM ≈ 0.96 and MSE ≈ 0.002 “(Table 1)”. In Table 1 those are the identity baseline’s numbers. BrLP’s are 0.930/0.936 and 0.0050/0.0041.

(b) There is no control for the autoencoder and segmentation pipeline:

  • The identity and random baselines are real scans. BrLP outputs are VAE-decoded synthetic images.
  • A segmenter applied to smoother synthetic images can shift CSF and tissue boundaries systematically.
  • A near-constant 6-8% CSF increase that barely depends on Δt\Delta t (R2R^2 = 0.047 and 0.089) looks more like that kind of offset than a learned progression prior. The same confound would inflate BrLP’s volumetric MAE.
  • The obvious control is an “identity through the autoencoder” baseline: encode and decode the baseline scan, segment it, and compare. If that reproduces the CSF shift, the conclusion about BrLP’s directional bias changes.

(c) The FID/KID/WASABI rows of Table 1 say nothing about task performance.

  • Every baseline in that table is a real scan, so those rows are real-vs-real comparisons. Real scans will win on any distributional metric by construction.
  • The Discussion’s claim that FID/KID “tracked the failures identified through other means” therefore does not follow from this comparison.

(d) BrLP’s auxiliary volume predictor and the LDM are never evaluated separately, so it is unclear which component the failure belongs to.

  1. The fMRI evidence has reporting inconsistencies and lacks the controls the paper itself recommends.

(a) LCFV is reported inconsistently:

  • Sec. 3.2.1 defines LCFV as log⁡(1−CFV)\log(1-\mathrm{CFV}). Fig. 2 behaves accordingly: lower is better, around -11 for data plus noise at low σ\sigma.
  • Table 2’s LCFV values are ln⁡(CFV)\ln(\mathrm{CFV}) instead: ln⁡(0.9492)=−0.052\ln(0.9492) = -0.052 and ln⁡(0.9450)=−0.057\ln(0.9450) = -0.057, matching the reported -0.0522 and -0.0568. Under the stated definition, NT-ViT would sit near -3.0 and -2.9.
  • The text (l. 321) gives a third value, LCFV ≈ -0.5. Fig. 3 (i, j) marks LCFV as higher-is-better.
  • This paper’s argument is that metric computation matters, and its AI statement says evaluation code was partly generated. That calls for an audit of all metric implementations, not a typo fix.

(b) Table 2 reports NT-ViT with no baseline or positive control:

  • There is no temporal-mean or training-mean volume baseline, no shuffled or time-shifted EEG control, and no positive control on Oddball/NODDI.
  • A positive control would be ground truth plus small noise, run through the same ROI pipeline on these datasets (not HCP). Without it, one cannot tell how much of ρ≈0\rho \approx 0 comes from the model versus the pipeline or the replication.
  • The R2R^2 values (-78 to -115) sit next to ROI MSE of about 0.004-0.007. That combination suggests ground-truth ROI temporal variance on the order of 10−510^{-5}. R2R^2 is then dominated by any mean offset and adds little beyond Pearson.

(c) The tokenizer study reports an anomaly as an advantage:

  • In Fig. 3k-l, MinMax reconstructions are classified more accurately (≈0.93-0.96) than the original data (dashed line, ≈0.84).
  • A reconstruction being more task-informative than its input is, by this paper’s own logic, a warning sign: leakage, or a scaling artifact the classifier exploits. It is presented as a strength of MinMax and not discussed.
  • SSIM and MSE are also compared across MinMax and Diff scalings. These live on different intensity scales unless the scaling is inverted before scoring.
  1. The broader message is also not very new. The inadequacy of SSIM/PSNR for subtle medical signals and the need for no-change baselines are familiar points, and Calhas & Henriques (2020), cited here, already proposed time-sensitive metrics for EEG-to-fMRI synthesis. For a paper of this kind, the evidence has to be cleaner than the evaluations it criticizes. At the moment it is not.
Questions for authors
  1. Which ADNI subjects were evaluated, and do any overlap with BrLP’s training split? 2,173 subjects is a large fraction of ADNI. Overlap would favor BrLP, but it changes how the ADNI results should be read.

  2. Which tool produced the ROI volumes, and was it the same for real and generated scans? Can you add the encode–decode baseline described in W1(b), at least for CSF and lateral ventricles?

  3. What is the denominator of “relative volumetric MAE”? Hippocampal MAEs of 0.01–0.03% (Fig. 1C–D) are far below scan–rescan variability if normalized by the ROI’s own volume. I assume it is a fraction of intracranial or brain volume; please state it.

  4. ADNI has 4,467 targets from 2,173 subjects. Do the Wilcoxon and McNemar tests account for repeated targets per subject? What is nn for the Δt≥10\Delta t \ge 10 yr subset (ADNI only, since OASIS-2 stops at 6 yr)?

  5. When a subject has several conditioning visits, which scan is the identity prediction: the last one?

  6. Which LCFV definition was used in Table 2, Fig. 2, Fig. 3 and the text respectively?

  7. Why use covariance rather than correlation matrices for the LE distance? Covariance makes LE sensitive to amplitude. In Fig. 2, the mean+noise curve is non-monotone, starting near 40 at σ=10−2\sigma = 10^{-2} and dipping to about 10 before rising. How many ROIs were used per atlas? What regularization did you apply when the number of ROIs approaches the number of frames (170 per Oddball run)?

  8. In the simulation, was z-scoring global or voxel-wise? Was SSIM computed per 3D frame and averaged, or on the 4D volume? What data range was used?

  9. Were the tokenizer and the logistic regression evaluated on held-out subjects? Randomized ordering across subjects (l. 910–911) does not by itself prevent leakage. How many subjects and runs were used, and how many seeds? Why do MinMax reconstructions exceed original-data accuracy?

  10. For NT-ViT, did you run a shuffled or time-shifted EEG control, or a positive control on the same data?

Reviewer_gw5hOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that conventional image-quality metrics, including SSIM, MSE, PSNR, FID, and KID, can misrepresent the performance of neuroimaging generative models by rewarding shared anatomical structure while overlooking scientifically meaningful signals. The authors re-evaluate two published models, BrLP for longitudinal structural MRI generation and NT-ViT for EEG-conditioned fMRI generation, alongside controlled fMRI simulations and tokenizer experiments. Their results show that favorable image-level scores can coexist with failures to capture disease progression or temporal dynamics, highlighting the need for task-specific evaluation designed to test model claims against appropriate baselines.

Critical strengths
  1. The paper highlights a meaningful evaluation failure mode in neuroimaging generative models, where strong image-level similarity can mask poor task performance.
  2. The identity, random-matching, XGBoost, and naive progression baselines make the limitations of conventional metrics easy to interpret.
  3. The fMRI simulations directly show that SSIM and MSE can remain favorable even after temporal information is removed.
Critical weaknesses
  1. The novelty of the proposed evaluation framework should be clarified. Several prior neuroimaging generation studies already use task- or domain-specific evaluation, including voxel-based morphometry, WASABI, diagnostic accuracy, and classifier-based verification. The contribution needs to be clarified and this distinction should be positioned more clearly relative to other prior work.
  2. The empirical conclusions are based primarily on two published generative models. Although these case studies span structural and functional imaging and are complemented by controlled experiments, it remains unclear how broadly the observed failure modes generalize across other generative tasks, architectures, and imaging modalities.
  3. The proposed falsification-oriented evaluation remains somewhat task dependent. The paper provides good examples of appropriate metrics and baselines for longitudinal structural MRI and fMRI generation, but still unclear how researchers should systematically determine the appropriate falsification tests, metrics, and trivial baselines for their neuroimaging generation task. Would be good to provide more practical and specific insights.
  4. The tokenizer experiments are informative but somewhat less directly connected to the central claim than the BrLP and NT-ViT case studies.
Questions for authors
  1. How do the authors distinguish the methodological novelty of the proposed framework from prior work that already evaluates generated neuroimaging using domain-specific measures such as roi-based volume, voxel-based morphometry, WASABI, diagnostic accuracy, or classifier-based verification?
  2. Can the authors provide more explicit guidance for selecting task-specific falsification tests and baselines for new applications? For example, what properties should a suitable trivial baseline satisfy?
  3. How broadly do the authors expect the demonstrated failure mode to apply? Would the same concern hold for generative tasks involving large focal abnormalities such as tumors or lesions, where the relevant signal contributes substantially more to image-level variation?
  4. Given that the main empirical evidence comes from BrLP and NT-ViT, will other additional generative models or modalities were showing similar issues?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.