- Rating
- 3(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies an important and under-examined question in empirical scaling: for hierarchically and spatially structured data, what does it actually mean to ``add more data''? Rather than representing data scale by a single sample count, the paper decomposes it into three axes: the number of unique samples , the number of contributing sources , and spatial/domain coverage . The study instantiates this decomposition in whole-brain histology, where sources are individual human brains and samples are spatially anchored image patches. Across 93 controlled pretraining runs using 11.6M candidate sample locations from 21 brains, the authors vary data allocation, compute, model capacity, spatial coverage, and cross-subject contributions to a SpatialNCE objective.
The central empirical result is both clear and somewhat surprising: inter-subject variation strongly matters for generalization---models perform substantially better on subjects encountered during pretraining---yet, at a fixed budget of roughly 450k unique patches, redistributing those patches over more subjects does not yield a detectable improvement in unseen-subject representation quality. In contrast, increasing the number of unique samples, compute, model capacity, and sufficiently broad spatial coverage generally improves representation quality.
My one-sentence summary of the contribution is: the paper shows that, in spatially structured representation learning, sample count, source diversity, and spatial coverage are distinct scaling variables, and that substantial source-level variation does not imply that a fixed sample budget should necessarily be distributed across more sources.
I find this conceptual distinction useful, memorable, and broader than the specific histology application. The problem is important, the central insight is substantive rather than a complicated recipe, and the experiments are unusually carefully controlled for a scaling study centered partly around a null result.
Critical strengths
- A clean and consequential conceptual decomposition of data scaling.
The strongest contribution is the framing of data acquisition as an allocation problem over unique samples, sources, and spatial coverage. This is more than a terminology change: the experiments explicitly manipulate these quantities instead of conflating them into a single notion of dataset size. This enables the paper to answer a practically meaningful question that conventional scaling experiments cannot answer, namely whether a fixed acquisition budget is better spent on denser sampling of existing sources or broader sampling across sources.
- The key fixed-budget result is supported by unusually extensive controlled evidence.
The main result is not inferred from uncontrolled comparisons across different datasets or models. In the key experiment, the unique-sample budget is held near 450k while the number of subjects is varied from 1 to 18, with matched presentation budget and broad anatomical coverage. The paper further examines model capacity, compute, spatial coverage, cross-subject contributions to the objective, and alternative held-out-subject assignments. The authors also quantify confidence intervals and minimum detectable effects rather than treating a nonsignificant difference as proof of equality. This makes the null result substantially more convincing than a standard ``no significant difference'' observation.
Critical weaknesses
- ``Source diversity'' is not completely separable from the induced SpatialNCE supervision graph.
At fixed , increasing the number of subjects does not only change biological source diversity. Because SpatialNCE defines supervision through spatial proximity, it also changes the composition of the positive-pair graph substantially. The total positive weight remains approximately constant in the main fixed-budget experiment, but the fraction contributed by cross-subject pairs increases from 0 to roughly 94% at .
This leaves a plausible alternative explanation for the flat source-scaling curve: additional subjects may provide useful biological diversity, while at the same time cross-subject anatomical correspondences may be more difficult or noisy, so that these effects partially cancel. Therefore, I believe the strongest justified conclusion is that \emph{in the tested SpatialNCE regime}, increasing the number of subjects at fixed patch budget provides no detectable gain. A stronger method-independent conclusion that source diversity itself has no marginal value would not yet be established.
I do not view this as fatal because the manuscript is already reasonably careful about restricting the conclusion to the tested regime, and the broader conceptual contribution---that samples, sources, and spatial coverage are different scaling axes---does not depend on resolving this mechanism.
- The empirical conclusion is necessarily narrow in task and objective space.
The fixed-sample source-scaling experiment is conducted at one main sample budget of roughly 450k unique locations, over --, using SpatialNCE and cytoarchitectonic-area classification with a linear probe. Source diversity may behave differently under substantially smaller data budgets, more sparse sampling per subject, other objectives, or downstream tasks whose relevant variation is less tightly coupled to anatomy.
I regard this as an external-validity limitation rather than a rejection-level flaw. Given the ICLR page limit, I would not require several additional benchmarks or SSL objectives for acceptance, but the paper should consistently preserve the scope of the claim.
Questions for authors
-
In the fixed-sample experiment, the total positive weight remains approximately constant, while the fraction arising from cross-subject pairs changes from 0% to about 94%. Do the authors intend the main conclusion to be specifically that ``source count has no detectable benefit under the SpatialNCE supervisory structure used here,'' or as a stronger statement about source diversity itself? Do you have any existing analysis that can distinguish a genuine absence of source-diversity benefit from a cancellation between additional biological diversity and changed or noisier cross-subject supervision?
-
SpatialNCE supervision is constructed from sample coordinates after mapping into Colin27 reference space, while the downstream task predicts Julich cytoarchitectonic areas. Can the authors explicitly confirm that the reference-space registration or coordinate construction used during pretraining does not use the downstream cytoarchitectonic labels, or information effectively equivalent to those labels? I do not currently see evidence of leakage, but because both the pretraining signal and downstream task are strongly anatomical, I think this point is important to rule out explicitly.
I would regard evaluation on additional unrelated downstream tasks or alternative SSL objectives as useful future work rather than necessary experiments for the present submission.
3 comments
Thank you for the detailed response and clarifications. After considering the rebuttal and the broader discussion, I have reconsidered my initial assessment.
While I still appreciate the careful experimental design and the scale of the study, I now believe I placed too much weight on these strengths relative to the limitations of the central finding. In particular, the authors acknowledge that the current experiments cannot distinguish a genuine lack of benefit from source diversity from a possible cancellation effect arising from changes in cross-subject supervision. Together with the reliance on a single pretraining objective and downstream task, this limits the generalizability and practical implications of the conclusions more than I initially appreciated.
I therefore decided to lower my rating. I still consider the work technically solid and the research question worthwhile, but my revised assessment reflects a more cautious view of the significance of its contribution.
Thank you for the thoughtful follow-up. We would like to add one point regarding the cancellation hypothesis.
Cancellation could explain why distributing a fixed sample budget across more brains produced no detectable gain. However, this would not change the observed net effect. Under the tested SpatialNCE setting, allocating the same number of samples across more brains did not detectably improve performance on unseen brains. The possible cancellation affects the interpretation of the underlying mechanism, but not the observed outcome of this allocation strategy.
The revised manuscript is now uploaded. Section 4.2 explicitly discusses your cancellation explanation alongside the fixed-budget result.
Thank you for the careful reading. Your summary is in line with our own perspective.
Is the conclusion specific to the SpatialNCE supervisory structure? Yes. At about 450,000 unique locations and one to 18 brains, spreading the patch budget over more brains gave no detectable gain under SpatialNCE with frozen-encoder area classification. Variation across brains may well carry value that this objective does not exploit. Smaller budgets or sparser sampling per brain may also behave differently. We held the objective, task, and readout fixed on purpose, so that the allocation question could be studied in depth. The same matched design can serve as a template for future studies with other objectives. The abstract and parts of Sections 1 and 5 state the result more broadly than this. We will restrict them to the tested setting.
Can existing analyses separate absent diversity benefit from cancellation? No. Your explanation is plausible: as brain count grows, the cross-brain share of positive weight rises, so useful variation could be offset by less reliable correspondence. Subject gating cannot separate the two, because it removes cross-brain positives while keeping cross-brain samples as negatives. A method that separates them would also be valuable for neuroscience, because it would quantify genuine individual variation. Indirect support for a role of subject variation itself comes from a similar drop on unseen brains under supervised training from scratch and supervised contrastive training, which use no coordinate-based correspondences (Schiffer et al., 2021; Schiffer et al., 2025). We will place your explanation next to the fixed-budget result in Section 4.2. We agree that other objectives, tasks, and adaptation are future work.
Do area labels or equivalent information enter registration or coordinate construction? No. Cytoarchitectonic labels, area boundaries, and maps derived from them do not enter the registration or the pretraining coordinates. Following Amunts et al. (2020, Supplementary Materials), each section is mapped by intensity-based linear and nonlinear registration to an MRI-guided reconstruction of its brain and then to the Colin27 template, an anatomical MRI. The Julich-Brain atlas later uses the same transformations to place its delineations in reference space. The delineations do not inform those transformations. The coordinates define SpatialNCE targets and are never encoder inputs. Anatomical location carries information about areas by design, which is what makes spatial supervision useful. No area-derived quantity is used to obtain it. Pretraining sections are subsampled by section position. Annotation status decides only which brains and sections serve for probing and evaluation. The three primary evaluation brains contribute neither pretraining data nor probe labels. We will state this separation in Section 3.1 and Appendix A.2.
References