acceptodds
Withdrawn from ICLR 2027

Cross-Sectional Accuracy Does Not Certify Within-Patient Tracking in Functional Connectivity

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Resting-state functional connectivity supports two applications at once, classifying disease and identifying individuals from their connectivity patterns. A profile carries two kinds of feature, those that stay fixed within a person and differ between people, and those that change within a person from visit to visit. Which kind carries the separable disease information has not been asked, yet the answer decides what a connectivity marker can be used for. We first localise that information to the fixed component, which reads the diagnosis at against from the whole profile, and the localisation holds under seven alternative removal methods. We then show that centring a profile on its subject mean cancels the class-mean difference exactly whenever the label is constant within a subject, so a linear readout of the residual is at chance by construction and cannot settle the question, while a label that does vary within a subject is retained. The test that remains is whether change within a person carries the label, which we run directly on subjects with repeated Mini-Mental State Examination scores, where the subject mean predicts the level at a Spearman correlation of while the within-subject rate of change predicts the rate of cognitive change at , against a detection limit of . The same placement further recurs on four of six cohorts spanning five disorders. A cross-sectional accuracy therefore certifies that patients differ from one another, not that connectivity detects change within one patient, and a marker meant to follow one patient has to be validated on within-subject change.

Primary area:
applications to neuroscience & cognitive science
Keywords:
functional connectivity, connectome fingerprinting, subject-mean centring, biomarker validation, longitudinal neuroimaging, within-subject change, invariant representations, Alzheimer's disease

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_rHGsOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper examines whether resting-state functional connectivity (FC) models that distinguish diagnostic groups can also track changes within individuals. The authors separate FC profiles into subject means and within-subject residuals, compare diagnostic and identity-related features, and test several representation-removal methods. On ADNI multi-visit data, diagnostic AUCs are 0.734 for full profiles, 0.728 for subject means, and 0.495 for residuals. Scan-level splitting also yields a much higher AUC than subject-level splitting (0.933 vs. 0.742).

Critical strengths
  1. Careful evaluation of data leakage. The subject-grouped cross-validation and permutation tests address an important issue in repeated-scan neuroimaging. The large gap between scan-level and subject-level AUCs (0.933 vs. 0.742) clearly shows how improper splitting can inflate diagnostic performance.
  2. Clear distinction between mathematical cancellation and longitudinal tracking. The paper recognizes that centering removes additive effects of subject-constant labels by construction, so chance-level residual classification alone cannot rule out longitudinal information. The controls support this distinction: centering improves head-motion prediction (0.577 to 0.708) and speech-emotion recognition (0.840 to 0.880), while reducing speaker-sex classification to chance (0.980 to 0.499).
Critical weaknesses
  1. Limited evidence for the longitudinal conclusion. The main tracking experiment uses only 57 ADNI participants, with high-dimensional FC slopes estimated from varying numbers of visits. The negative result (ρ = −0.186, 95% CI [−0.514, 0.046]) shows that this pipeline fails to predict MMSE change, but does not rule out longitudinal information detectable by other approaches. More importantly, Appendix E.3 reports that simulated temporal trends were not detected at the tested strengths, leaving the method's sensitivity to longitudinal changes unclear. The reported fivefold detection threshold concerns variance differences, not temporal trends. The structural MRI comparison does not fully address this issue either: while structural rates predict diagnosis (AUC = 0.807), they also fail to predict MMSE change (ρ = −0.084, n = 55; Table 13). This raises concerns about whether the evaluation can reliably detect cognitive decline. The longitudinal conclusions should therefore be more cautious.
  2. Unclear distinction between subject identity and disease-related information. Centering removes additive effects of subject-constant labels by construction, but this does not imply that disease-related longitudinal changes are absent. The seven removal methods also lack a consistent evaluation of identity suppression. For instance, pulling visits from the same subject together through contrastive learning may preserve identity information rather than remove it. Appendix E.3 further shows that residuals retain substantial subject-identification accuracy (0.245 vs. 1/61 chance). The single-visit subspace analyses cannot establish temporal stability, and the ADNI subspace-angle result is not significant (p = 0.312; Table 15). Finally, Proposition 4 assumes an unquantified bound on residual identity information, and its data-processing argument needs clarification since centering uses all visits rather than a single observation. These issues weaken the proposed interpretation of where disease information resides.
Questions for authors
  1. Reliability of longitudinal estimates. How reliable are the FC and MMSE slopes across different numbers of visits and follow-up durations? How sensitive are the results to age, sex, acquisition site, and motion? Also, why do structural MRI change rates predict diagnosis (AUC = 0.807) but fail to predict MMSE change?
  2. Sensitivity to temporal trends. Appendix E.3 reports that injected temporal trends were not detected at the tested strengths. What is the minimum temporal effect size the slope-based analysis can reliably detect? A simulation or power analysis accounting for uncertainty in individual slope estimates would be more informative than the Fisher-z threshold alone.
  3. Identity removal and theoretical assumptions. Can the authors measure identity predictability consistently before and after each of the seven removal methods? Are held-out subject means computed using all visits, including future observations relative to the prediction time? Finally, can the authors clarify the data-processing argument in Proposition 4 and justify the assumed bound on residual identity information?
Reviewer_dFxPOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper asks whether disease information in resting-state functional connectivity lies in the between-subject (subject-mean) component or in the within-subject component. On ADNI, the authors find that the diagnosis is carried by the subject mean, and that within-subject changes in connectivity do not track changes in cognition. From this, they conclude that cross-sectional accuracy does not certify within-patient tracking.

Critical strengths

The research question is relevant. Whether a marker validated cross-sectionally can track change within a patient matters for the clinical use of connectivity biomarkers.

Critical weaknesses
  1. In my opinion, the study is suitable to submit to neuroimaging journal rather than fitting for ICLR. The methods are standard statistics, with linear classifiers, centring and permutation tests, and the neural models serve only as side checks. More importantly, the study offers no new learning method, representation, or insight for the ML community.
  2. The writing is dense and repetitive, which makes the actual contributions hard to identify. AUC values are reported as “accuracy” in the abstract. Some highlighted differences are far below the fold-level variability: 0.728 vs. 0.734 against a standard deviation of about 0.12.
  3. The abstract and contributions say the result “recurs on four of six cohorts.” These cohorts are single-session, so they cannot test within-subject tracking at all, and only a subspace-angle measure is reported for them. On ADNI, the primary cohort, the angle is not significant (p=0.312).
  4. A disease effect on a subject-constant label raises that edge’s between-subject variance, and hence ReR_e, by definition. The label-permutation null therefore cannot separate “disease information rides on identity edges” from “disease is a between-subject effect.” The evidence is also weak: p=0.215 at k=100, a ratio of only 1.3 at k=5000, and a parcel-level overlap of p=0.081. In addition, Welch t-statistics computed across scans treat repeated scans from the same subject as independent.
Questions for authors

What is the intended contribution to the machine learning community beyond the neuroimaging validation message?

Reviewer_CC1uOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to investigate whether diagnostic information in resting-state functional connectivity primarily comes from inter-individual differences or changes within individuals over time. It decomposes functional connectivity into subject means and within-subject residuals, and then examines how diagnostic information is distributed across these components using different methods across cohorts. In the experiment, subject-mean features achieved diagnostic performance close to that of the full FC profiles, but within-subject changes in FC did not reliably predict cognitive change. Based on these findings, the authors emphasize that cross-sectional classification performance cannot replace direct evaluation of a model’s ability to track changes in an individual’s condition. Overall, the research problem is of interest and it performs a broad range of control experiments.

Critical strengths
  1. The research topic is interesting. This work tries to investigate how functional connectivity biomarkers should be used. The authors also test longitudinal prediction by using rates of change in functional connectivity to predict rates of change in cognitive scores. This provides useful guidance for evaluating and applying neuroimaging models in clinical studies.
  2. The experiments cover different modalities and analysis methods. It compares raw functional connectivity, subject means, and residuals, with additional analyses involving head motion, different processing methods, structural MRI, and simulated signals. The appendix further examines diagnostic information in the residuals using kernel classifiers, gradient boosting, and measures of variance, magnitude, slopes, and lagged correlations.
Critical weaknesses
  1. The decoupling and roles of different connectivity features have long been studies. The novelty and contributions of this work are still limited to the general area.
  2. The related work is limited. The most recent references in the paper date back to 2022. However, the recent studies related to decoupling of individual features and disease-specific features, and the comparison between cross-sectional prediction and longitudinal tracking using functional connectivity are missing.
  3. The cognitive change prediction analysis includes only 57 participants. Although conducting statistical tests and power analysis, the sample may still be too small to detect weaker associations.
  4. Some expressions are wordy and uncommon in this field, making it hard to follow.
Questions for authors
  1. Section 3.3 uses 20 dimensions for the subspace for capturing between-subject differences and 10 dimensions for the diagnosis-related subspace. How were these dimensions chosen?
  2. Table 13 shows that, in the same group of 55 participants, both structural MRI and FC rates of change fail to predict MMSE rates of change. Could this suggest that the prediction failure also reflects limitations of the data or the task itself, rather than limitations of FC alone?
  3. The multi-visit cohort had 2 to 9 visits per participant. Estimates of FC and MMSE slopes obtained using ordinary least squares may be affected by the number of visits and the length of follow-up.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.