acceptodds
Under review as a conference paper at ICLR 2027

Samples, Sources, Space: Decomposing Data Scale in Spatially Structured Representation Learning of Human Brain Microarchitecture

Abstract

Scaling studies typically represent training data by a single count of samples. For hierarchically and spatially structured data, however, the same number of samples can be drawn from few or many sources and distributed differently across the underlying domain. We therefore study data scaling as an allocation problem, separating unique sample count, source diversity, and spatial coverage. We study this decomposition in microscopic whole-brain histology, where a source is an individual brain, and a sample is an image patch at a specific spatial location. Across 93 controlled pretraining runs of a contrastive model that uses spatial proximity for supervision, we vary data allocation, compute, and model capacity over 11.6 million spatially anchored image patches from 21 human brains. Performance improves with more unique samples, broader spatial coverage, additional compute, and larger model capacity. At fixed sample count, distributing samples across one to 18 subjects produces no detectable improvement, even though representations generalize substantially better to subjects encountered during pretraining. Inter-subject variation therefore strongly affects generalization, but additional subjects provide no benefit when a fixed sample budget is distributed across more sources. These results establish sample count, source diversity, and spatial coverage as distinct axes of data scaling in spatially structured representation learning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning, self-supervised learning, data scaling, dataset composition, scaling, spatial representation learning, computational neuroscience, human brain

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_U3D4OpenReview ↗
Rating
3(−1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies an important and under-examined question in empirical scaling: for hierarchically and spatially structured data, what does it actually mean to ``add more data''? Rather than representing data scale by a single sample count, the paper decomposes it into three axes: the number of unique samples NuniqueN_{\mathrm{unique}}, the number of contributing sources BB, and spatial/domain coverage ρ\rho. The study instantiates this decomposition in whole-brain histology, where sources are individual human brains and samples are spatially anchored image patches. Across 93 controlled pretraining runs using 11.6M candidate sample locations from 21 brains, the authors vary data allocation, compute, model capacity, spatial coverage, and cross-subject contributions to a SpatialNCE objective.

The central empirical result is both clear and somewhat surprising: inter-subject variation strongly matters for generalization---models perform substantially better on subjects encountered during pretraining---yet, at a fixed budget of roughly 450k unique patches, redistributing those patches over more subjects does not yield a detectable improvement in unseen-subject representation quality. In contrast, increasing the number of unique samples, compute, model capacity, and sufficiently broad spatial coverage generally improves representation quality.

My one-sentence summary of the contribution is: the paper shows that, in spatially structured representation learning, sample count, source diversity, and spatial coverage are distinct scaling variables, and that substantial source-level variation does not imply that a fixed sample budget should necessarily be distributed across more sources.

I find this conceptual distinction useful, memorable, and broader than the specific histology application. The problem is important, the central insight is substantive rather than a complicated recipe, and the experiments are unusually carefully controlled for a scaling study centered partly around a null result.

Critical strengths
  1. A clean and consequential conceptual decomposition of data scaling.

The strongest contribution is the framing of data acquisition as an allocation problem over unique samples, sources, and spatial coverage. This is more than a terminology change: the experiments explicitly manipulate these quantities instead of conflating them into a single notion of dataset size. This enables the paper to answer a practically meaningful question that conventional scaling experiments cannot answer, namely whether a fixed acquisition budget is better spent on denser sampling of existing sources or broader sampling across sources.

  1. The key fixed-budget result is supported by unusually extensive controlled evidence.

The main result is not inferred from uncontrolled comparisons across different datasets or models. In the key experiment, the unique-sample budget is held near 450k while the number of subjects is varied from 1 to 18, with matched presentation budget and broad anatomical coverage. The paper further examines model capacity, compute, spatial coverage, cross-subject contributions to the objective, and alternative held-out-subject assignments. The authors also quantify confidence intervals and minimum detectable effects rather than treating a nonsignificant difference as proof of equality. This makes the null result substantially more convincing than a standard ``no significant difference'' observation.

Critical weaknesses
  1. ``Source diversity'' is not completely separable from the induced SpatialNCE supervision graph.

At fixed NuniqueN_{\mathrm{unique}}, increasing the number of subjects does not only change biological source diversity. Because SpatialNCE defines supervision through spatial proximity, it also changes the composition of the positive-pair graph substantially. The total positive weight remains approximately constant in the main fixed-budget experiment, but the fraction contributed by cross-subject pairs increases from 0 to roughly 94% at B=18B=18.

This leaves a plausible alternative explanation for the flat source-scaling curve: additional subjects may provide useful biological diversity, while at the same time cross-subject anatomical correspondences may be more difficult or noisy, so that these effects partially cancel. Therefore, I believe the strongest justified conclusion is that \emph{in the tested SpatialNCE regime}, increasing the number of subjects at fixed patch budget provides no detectable gain. A stronger method-independent conclusion that source diversity itself has no marginal value would not yet be established.

I do not view this as fatal because the manuscript is already reasonably careful about restricting the conclusion to the tested regime, and the broader conceptual contribution---that samples, sources, and spatial coverage are different scaling axes---does not depend on resolving this mechanism.

  1. The empirical conclusion is necessarily narrow in task and objective space.

The fixed-sample source-scaling experiment is conducted at one main sample budget of roughly 450k unique locations, over B=1B=1--1818, using SpatialNCE and cytoarchitectonic-area classification with a linear probe. Source diversity may behave differently under substantially smaller data budgets, more sparse sampling per subject, other objectives, or downstream tasks whose relevant variation is less tightly coupled to anatomy.

I regard this as an external-validity limitation rather than a rejection-level flaw. Given the ICLR page limit, I would not require several additional benchmarks or SSL objectives for acceptance, but the paper should consistently preserve the scope of the claim.

Questions for authors
  1. In the fixed-sample experiment, the total positive weight MM remains approximately constant, while the fraction arising from cross-subject pairs changes from 0% to about 94%. Do the authors intend the main conclusion to be specifically that ``source count has no detectable benefit under the SpatialNCE supervisory structure used here,'' or as a stronger statement about source diversity itself? Do you have any existing analysis that can distinguish a genuine absence of source-diversity benefit from a cancellation between additional biological diversity and changed or noisier cross-subject supervision?

  2. SpatialNCE supervision is constructed from sample coordinates after mapping into Colin27 reference space, while the downstream task predicts Julich cytoarchitectonic areas. Can the authors explicitly confirm that the reference-space registration or coordinate construction used during pretraining does not use the downstream cytoarchitectonic labels, or information effectively equivalent to those labels? I do not currently see evidence of leakage, but because both the pretraining signal and downstream task are strongly anatomical, I think this point is important to rule out explicitly.

I would regard evaluation on additional unrelated downstream tasks or alternative SSL objectives as useful future work rather than necessary experiments for the present submission.

3 comments

Thank you for the careful reading. Your summary is in line with our own perspective.

Is the conclusion specific to the SpatialNCE supervisory structure? Yes. At about 450,000 unique locations and one to 18 brains, spreading the patch budget over more brains gave no detectable gain under SpatialNCE with frozen-encoder area classification. Variation across brains may well carry value that this objective does not exploit. Smaller budgets or sparser sampling per brain may also behave differently. We held the objective, task, and readout fixed on purpose, so that the allocation question could be studied in depth. The same matched design can serve as a template for future studies with other objectives. The abstract and parts of Sections 1 and 5 state the result more broadly than this. We will restrict them to the tested setting.

Can existing analyses separate absent diversity benefit from cancellation? No. Your explanation is plausible: as brain count grows, the cross-brain share of positive weight rises, so useful variation could be offset by less reliable correspondence. Subject gating cannot separate the two, because it removes cross-brain positives while keeping cross-brain samples as negatives. A method that separates them would also be valuable for neuroscience, because it would quantify genuine individual variation. Indirect support for a role of subject variation itself comes from a similar drop on unseen brains under supervised training from scratch and supervised contrastive training, which use no coordinate-based correspondences (Schiffer et al., 2021; Schiffer et al., 2025). We will place your explanation next to the fixed-budget result in Section 4.2. We agree that other objectives, tasks, and adaptation are future work.

Do area labels or equivalent information enter registration or coordinate construction? No. Cytoarchitectonic labels, area boundaries, and maps derived from them do not enter the registration or the pretraining coordinates. Following Amunts et al. (2020, Supplementary Materials), each section is mapped by intensity-based linear and nonlinear registration to an MRI-guided reconstruction of its brain and then to the Colin27 template, an anatomical MRI. The Julich-Brain atlas later uses the same transformations to place its delineations in reference space. The delineations do not inform those transformations. The coordinates define SpatialNCE targets and are never encoder inputs. Anatomical location carries information about areas by design, which is what makes spatial supervision useful. No area-derived quantity is used to obtain it. Pretraining sections are subsampled by section position. Annotation status decides only which brains and sections serve for probing and evaluation. The three primary evaluation brains contribute neither pretraining data nor probe labels. We will state this separation in Section 3.1 and Appendix A.2.

References

  • Amunts, K., Mohlberg, H., Bludau, S., & Zilles, K. (2020). Julich-Brain: A 3D probabilistic atlas of the human brain's cytoarchitecture. Science, 369(6506), 988–992.
  • Schiffer, C., Amunts, K., Harmeling, S., & Dickscheid, T. (2021). Contrastive representation learning for whole brain cytoarchitectonic mapping in histological human brain sections. IEEE ISBI 2021, 603–606.
  • Schiffer, C., et al. (2025). CytoNet: A foundation model for the human cerebral cortex at cellular resolution. arXiv:2511.01870.

Thank you for the detailed response and clarifications. After considering the rebuttal and the broader discussion, I have reconsidered my initial assessment.

While I still appreciate the careful experimental design and the scale of the study, I now believe I placed too much weight on these strengths relative to the limitations of the central finding. In particular, the authors acknowledge that the current experiments cannot distinguish a genuine lack of benefit from source diversity from a possible cancellation effect arising from changes in cross-subject supervision. Together with the reliance on a single pretraining objective and downstream task, this limits the generalizability and practical implications of the conclusions more than I initially appreciated.

I therefore decided to lower my rating. I still consider the work technically solid and the research question worthwhile, but my revised assessment reflects a more cautious view of the significance of its contribution.

Thank you for the thoughtful follow-up. We would like to add one point regarding the cancellation hypothesis.

Cancellation could explain why distributing a fixed sample budget across more brains produced no detectable gain. However, this would not change the observed net effect. Under the tested SpatialNCE setting, allocating the same number of samples across more brains did not detectably improve performance on unseen brains. The possible cancellation affects the interpretation of the underlying mechanism, but not the observed outcome of this allocation strategy.

The revised manuscript is now uploaded. Section 4.2 explicitly discusses your cancellation explanation alongside the fixed-budget result.

Reviewer_pbkXOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper motivates an analysis of how different factors affect the classification of whole brain histology using representations learned from existing data, and how these representations generalize to new data from the same subjects or to histology from new subjects. It studies how sample count, subject diversity, and spatial coverage affect representation learning. The authors conduct different controlled pretraining runs using SpatialNCE, a contrastive objective based on anatomical proximity, with a dataset of 11.6 million image patches from 21 human brains. They evaluate the representations through linear probing for cytoarchitectonic area classification. They report benefits from more unique samples, broader spatial coverage, additional compute, and larger models, while distributing a fixed sample budget across more subjects provides no detectable benefit in the tested setting.

Critical strengths
  1. Relevant research question. Optimizing which data to collect is important, particularly when acquiring additional subjects is costly. The paper’s attempt to distinguish the value of more samples, more subjects, and broader anatomical coverage addresses a relevant question, even though the practical conclusions remain insufficiently established.
Critical weaknesses
  1. Unclear practical contribution and limited new insight. The main motivation is to guide data acquisition, but there is a substantial gap between this motivation and the practical findings. Several observations, including better performance with more samples, compute, model capacity, and subject familiarity, are expected and have been established in many other tasks. The main takeaway appears to be that more data helps, without clearly establishing which additional data are most valuable or easiest to acquire. The comparisons at a fixed sample budget do not sufficiently resolve these acquisition decisions, particularly when acquiring more subjects is itself a way to obtain more samples. Overall, I do not see a clear methodological contribution or sufficiently new and important understanding from the results. Questions 1 to 4 address this gap.

  1. Limited experimental scope and method comparisons restrict the conclusions. The experiments focus on one pretraining objective, one classification task, and primarily fully unseen subject evaluation, without investigating efficient adaptation to the target subject. It remains unclear whether the subject diversity findings reflect a general property of data allocation or the particular SpatialNCE objective and evaluation setup. Comparisons with alternative learning methods and adaptation strategies are needed to understand the scope of the conclusions. Unclear aspects of probe sampling and optimization further limit confidence in the comparisons. The technical presentation also makes the experimental intuition and takeaways difficult to follow. Questions 2 and 5 to 11 concern these limitations.

Overall Assessment. I consider this paper below the acceptance threshold. The results do not establish sufficiently new understanding or clear practical guidance for data acquisition, while the limited methods, tasks, and evaluation settings restrict the conclusions.

Questions for authors
  1. Subject familiarity. The advantage of including a subject in pretraining in Figure 2A and Section 4.1 is expected. What does this experiment reveal about how to improve generalization to new subjects?

  2. Dependence on the learning method. Could the lack of benefit from additional subjects at a fixed sample budget in Section 4.2 be specific to SpatialNCE? Since acquiring more subjects also provides more samples, how does this finding guide practical acquisition decisions?

  3. Novelty of the scaling findings. Section 4.3 shows benefits from more samples, compute, and model capacity, which are familiar from many other settings. What new, actionable insight does this analysis provide for data collection?

  4. Practical value of spatial coverage. Better performance from broader coverage is expected, and the experiment does not separate coverage from changes in spatial supervision. Does collecting additional patches from new regions provide greater benefit than sampling existing regions more densely? Without establishing this acquisition tradeoff, the practical contribution remains unclear.

  5. Consistency of probe training data. Were all linear probes trained on exactly the same image patches and labels across encoder configurations? Different training sets could confound the comparison between pretrained representations.

  6. Training randomness. How many optimization seeds were used for probe training, and how sensitive are the results to probe sampling and training randomness?

  7. Probe convergence and hyperparameters. Do all probes converge under the fixed schedule, and do the conclusions hold across different learning rates, regularization strengths, and training durations?

  8. Acquisition consistency. Were all subjects acquired under the same experimental setup, or could differences in tissue processing and imaging contribute to the subject effects?

  9. Anatomical alignment. How accurate is the registration across subjects, and could anatomical misalignment weaken the coordinate based supervision used by SpatialNCE?

  10. Adaptation to the target subject. Would the subject diversity findings change with limited encoder adaptation to the target subject, for example using LoRA, rather than fully unseen subject evaluation?

  11. Clarity of the results. Figure 2C combines several factors, while Section 4.4 and Figure 5 are difficult to follow. Could the comparisons and takeaways be explained more intuitively, with explicit descriptions rather than experiment identifiers such as E1 to E8? More generally, the results need clearer analysis connecting the technical details to the motivation.

4 comments

Thank you for the detailed questions. The two main concerns are the practical contribution (Q1–Q4) and the experimental scope (Q2, Q10). We will restrict the claims in the abstract and Sections 1, 4.2, 4.4, and 5 to the tested SpatialNCE setting.

Practical contribution (Q1–Q4). A large brain-specific effect might suggest spreading patches over as many brains as possible. Our results show that this does not follow when the number of patches is fixed.

  • Q1: What does subject familiarity reveal about generalization to new brains? The advantage itself is expected and was reported before (Schiffer et al., 2021; Schiffer et al., 2025, CytoNet). Previous work did not test whether scaling can replace it. The advantage grew with the number of patches per familiar brain and less with brain count (Section 4.1, Figure 2C, an association), so more data per brain did not remove it. A brain to be mapped should therefore be part of pretraining, which needs no area labels from it. In CytoNet, even a short continuation of self-supervised pretraining on a new brain improved results substantially. For brains that remain unseen, deeper sampling of the training brains helped more than adding brains.
  • Q2: How does the fixed-budget result guide acquisition when new brains also bring more patches? We agree that a new brain also brings more patches. Growing both together improved performance (Section 4.3). A newly acquired brain should therefore enter pretraining with its patches added to the budget, keeping the number of patches per brain. The fixed-budget result informs a second decision: how to use processed tissue. There, a larger pool of distinct patches helped. Trading patches for more brains gave no detectable gain. We will separate these two decisions in Section 5.
  • Q3: What is new and actionable in the scaling findings of Section 4.3? The direction of these trends is expected. What the design adds is their size relative to the brain-count effect, measured under the same controls. One might also expect more brains to require larger models. The brain-count effect did not grow detectably with capacity (Section 4.2), a null result we consider worth reporting. Part of the patch-pool gain may come from less repetition per patch.
  • Q4: Do patches from new regions help more than denser sampling of existing ones? At a fixed budget with 13 brains, concentrating the patches in a localized block of sections reduced performance compared with sampling every tenth section, in two of three conditions beyond the detection threshold (Section 4.4). For this budget, broad sampling outperformed dense local sampling. The localized conditions also change the SpatialNCE positive weight. In practice, choosing patches also fixes the supervision, so the combined effect is the relevant one. We will state in Section 4.4 that this evidence rests on exploratory contrasts and that coverage and supervision change together.

Scope (Q2, Q10): Is the result specific to SpatialNCE, one task, and frozen encoders? The fixed-budget result is established for SpatialNCE, the standard objective for this data (Schiffer et al., 2025). Other objectives may behave differently. The drop on unseen brains itself also appeared with supervised training from scratch and supervised contrastive training (Schiffer et al., 2021; Schiffer et al., 2025). Under SpatialNCE, adding brains also changes which pairs supervise training. We will name this in Section 4.2 as a possible reason for an objective-specific result. The present study alone required about 349,000 GH200 accelerator-hours. Crossing the allocation grid with further objectives, tasks, or adaptation methods would be prohibitive for one study. Reviewer U3D4 describes the resulting evidence as unusually extensive. We see the paper as a first report of these effects and as a template for such experiments: fix the number of unique samples and the compute, vary the number of sources over independent source subsets, and expand the sample pool within fixed sources.

Q5: Were all probes trained on the same patches and labels? Yes. Within each held-out-brain assignment, every probe reads one precomputed set of patches and labels. The alternative assignments (Appendix D.5) use their own fixed sets because the eligible brains differ. We will state this in Appendix A.3.

Q6: How many seeds were used, and how sensitive are the results to randomness? Each configuration was pretrained and probed with one seed. The probe protocol follows CytoNet, which reports small variation of linear-probe scores across seeds. Our initial tests confirmed this. We spent the compute on independent brain subsets, whose spread captures sensitivity to which brains are sampled. We will add this rationale to Appendix A.3.

Q7: Do all probes converge under the fixed schedule and hyperparameters? All probes share one schedule and one set of hyperparameters, derived from the CytoNet experiments (Schiffer et al., 2025). All probes converged under this schedule. In pilot runs, longer training gave diminishing gains without changing encoder rankings. We will add this to Appendix A.3.

Q8: Could differences in tissue processing and imaging contribute to subject effects? All brains followed the same processing and imaging protocol. Residual technical differences may still contribute (Schiffer et al., 2025). We therefore interpret subject effects as combining biological and technical variation.

Q9: How accurate is the registration, and could misalignment weaken SpatialNCE? Sections are registered by intensity-based linear and nonlinear registration through an MRI-guided reconstruction of each brain to the Colin27 template (Holmes et al., 1998). This is the established workflow of the Julich-Brain atlas, including its quality control (Amunts et al., 2020), and Colin27 is a standard MNI reference template. We also visually inspected the training locations in Colin27 space. Random patch translations during training (Appendix B.1) and a SpatialNCE kernel that is wide compared with the patch size make the supervision tolerant of small misalignments. We will expand the registration description in Appendix A.2.

Q10: Would limited adaptation to the target brain, such as LoRA, change the findings? Frozen encoders isolate what the pretraining data contribute. CytoNet showed that limited adaptation to a target brain improves results on that brain (Schiffer et al., 2025). Adaptation such as LoRA may also change the brain-count trend. Our goal is to understand how data allocation shapes pretrained representations, so we will name adaptation as future work in Section 5.

Q11: Can Figure 2C, Section 4.4, Figure 5, and the E1–E8 identifiers be explained more clearly? We will add descriptive names alongside E1–E8 where the page limit allows, state for each comparison in Section 4 what it holds fixed and why, and mark Figure 2C as an association across configurations.

References

  • Amunts, K., Mohlberg, H., Bludau, S., & Zilles, K. (2020). Julich-Brain: A 3D probabilistic atlas of the human brain's cytoarchitecture. Science, 369(6506), 988–992.
  • Holmes, C. J., et al. (1998). Enhancement of MR images using registration for signal averaging. J. Comput. Assist. Tomogr., 22(2), 324–333.
  • Schiffer, C., Amunts, K., Harmeling, S., & Dickscheid, T. (2021). Contrastive representation learning for whole brain cytoarchitectonic mapping in histological human brain sections. IEEE ISBI 2021, 603–606.
  • Schiffer, C., et al. (2025). CytoNet: A foundation model for the human cerebral cortex at cellular resolution. arXiv:2511.01870.

I thank the authors for their time and responses to each question. The responses clarified several aspects of the experimental setup, particularly Q5, Q6, Q7, and Q9, and I the planned presentation improvements in response to Q11. However, they did not resolve my main concerns.

While I believe the experiments were conducted properly within the scope of this work, I still find the practical contribution very limited. The finding that more distinct patches help, while distributing a fixed patch budget across more brains provides no detectable benefit under SpatialNCE, is a useful observation. However, I do not think it provides sufficiently new insight or translates into clear guidance for planning future experiments. My second main concern is that this conclusion may depend on how the method integrates subjects. Considerable effort in other fields goes into models and training procedures that exploit both shared information and individual differences. I would therefore expect targeted comparisons with approaches that account for these factors before drawing practical conclusions about the tradeoff between more samples and more subjects. I understand the computational constraints, but restricting the claims to SpatialNCE also substantially limits their implications.

My two main concerns therefore remain: Unclear practical contribution and limited new insight. The results do not establish sufficiently new understanding or clear practical guidance for planning future experiments. Limited experimental scope and method comparisons. It remains unclear whether the conclusions generalize beyond SpatialNCE or reflect how this particular method integrates data across subjects.

I again appreciate the authors’ time and detailed responses. I believe the research question is interesting and recognize the substantial, systematic work presented here. However, given the current experiments and conclusions, I unfortunately do not think the paper should be accepted.

Thank you for the follow-up and for confirming that the clarifications on the experimental setup addressed those questions. We appreciate your assessment and acknowledge the remaining concerns about scope and significance. The revised manuscript is now uploaded.

Reviewer_LEN9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on the specific aspect of data, within the context of scaling studies. Most existing scaling law studies take a simplified approach to data, with a single count of the amount of data seen during training. However, in many application domains, such as microscopy whole-brain histology as examined in this paper, the data has a hierarchical structure, where overall unique samples can be distributed across different subjects, and with differing spatial coverage. Thus there are multiple dimensions to consider where increasing data set size.

This paper formulates an experimental design that explicitly considers these different modes of variation. The paper finds, for their particular problem domain and data, that while inclusion of a source in pre-training has the largest observed effect on performance, for a fixed sample budget, distributing this sample budget across more subject sources does not give a significant improvement. In comparison, increasing unique sample count, and increasing spatial coverage under fixed sample budget, give statistically significant improvements.

Critical strengths

In general, this paper looks at an important practical question for many problem domains that have these underlying issues of data hierarchy and how to allocate training samples across different sources. The paper also gives a careful experimental study of this question for the problem of whole brain histology. The paper also gives an interesting finding that, for this particular set-up, adding more data sources does not seem to help at a fixed sample budget.

Critical weaknesses

Two potentially interrelated weaknesses:

(1) The paper findings are specific to the single problem domain studied. It is difficult then to extrapolate findings to another potential problem domain, other than replicating the entire experimental set-up for that problem domain.

(2) The core novelty of the paper is then in raising/formulating this question, and in the proposed experimental set-up, so overall the novelty is arguably somewhat minor.

The paper would potentially be significantly strengthened by applying the set-up to additional problem domains, and seeing if there are more general guidelines/take-aways. For instance, are there ways to characterize inter versus intra source diversity, to predict whether there is benefit to adding additional sources?

As an example of a recent related paper that gives a more prescriptive framework for improving data allocation without running a full set of scaling experiments, see Knowledge Transfer Scaling Laws for 3D Medical Imaging, to be presented at Neurips 2026.

Questions for authors

Nothing specific, outside of any responses to the raised weaknesses above.

1 comment

Thank you for the careful summary and the pointer to related work.

(1) Findings specific to one domain. The numerical results hold for whole-brain histology with SpatialNCE. The experimental design carries over to other domains. Matched comparisons need many complete pretraining runs at fixed compute and model capacity. Our 93 runs, up to ViT-H on 4096-token inputs, and their probes used about 349,000 GH200 accelerator-hours. Repeating this grid in several domains would multiply that cost and is beyond the scope of one paper. We therefore studied one domain in depth, so that others can build on it without repeating the full setup. In whole-brain histology, many large patches with known anatomical positions are nested in few brains, each of which is costly to process. This nesting is much stronger than in typical medical imaging, where a subject contributes one or a few scans. Cytoarchitectonic mapping is also a central problem in human brain research. The same structure appears in Earth observation, where image tiles are nested in acquired scenes and located in shared geographic coordinates. More data can then mean more tiles from existing scenes, more scenes, or broader geographic coverage. We will add this rationale and this example to Sections 1 and 5.

(2) Novelty. Scaling-law studies represent data by a single count (Kaplan et al., 2020; Hoffmann et al., 2022). Muennighoff et al. (2023) separate unique from repeated data, without considering sources. In pathology, the value of source diversity is inferred from comparisons across published models (Campanella et al., 2025; Bosch et al., 2025), where sources, samples, objectives, and compute all differ. We vary the number of sources at a fixed number of unique samples, with matched compute and capacity, independent source subsets, and three model sizes. This separates the effect of more sources from that of more samples, which cross-model comparisons cannot do. In our setting, a large advantage on brains seen in pretraining (Section 4.1) did not translate into a gain from more brains at a fixed budget (Section 4.2). More distinct patches from the same brains did help (Section 4.3).

Can inter- versus intra-source diversity predict the benefit of adding sources? Our results caution against one natural predictor. The familiar-versus-unseen gap directly measures inter-source variation and was large in our data. Yet it did not predict whether more brains help at a fixed budget. The gap was associated more with patches per familiar brain than with brain count (Section 4.1, Figure 2C). Any such predictor should therefore be checked against a matched comparison. We leave developing one to future work and will add this point to Section 5.

We will cite Lee et al. (2026) and make the link to our work explicit in Section 2. Their framework predicts allocation from estimated transfer between modality–anatomy domains. Our sources are individual brains within one modality. The two approaches complement each other.

References

  • Bosch, C., et al. (2025). Diversity over scale: Whole-slide image variety enables H&E foundation model training with fewer patches. arXiv:2511.10286.
  • Campanella, G., et al. (2025). A clinical benchmark of public self-supervised pathology foundation models. Nature Communications, 16, 3640.
  • Hoffmann, J., et al. (2022). Training compute-optimal large language models. arXiv:2203.15556.
  • Kaplan, J., et al. (2020). Scaling laws for neural language models. arXiv:2001.08361.
  • Lee, H. H., et al. (2026). Knowledge transfer scaling laws for 3D medical imaging. arXiv:2605.06859. To appear at NeurIPS 2026.
  • Muennighoff, N., et al. (2023). Scaling data-constrained language models. NeurIPS 2023.

Comments on the paper

1 comment

We thank all three reviewers for their careful reading. Two concerns recur across the reviews: what a study in one domain with one objective can teach beyond that domain, and what practical guidance it gives. We reply now to open the discussion early. The revised manuscript will follow during the discussion period. Each reply names the changes we plan. In our replies, "brains" are the subjects of the paper.

Why this study and this domain. Controlled data-allocation studies at the scale of current vision encoders are rarely feasible. Each comparison needs many full pretraining runs at matched compute and model capacity. A dedicated allocation on a publicly funded supercomputer allowed us to run 93 such runs from ViT-B to ViT-H. Together with their probes, they used about 349,000 GH200 accelerator-hours. Reporting an experiment of this size in full spares others from repeating it and gives them a design to reuse with other objectives and datasets. Whole-brain histology is a representative case for this design. Each brain yields hundreds of thousands of patches with known anatomical positions. Adding a brain means processing an entire postmortem specimen.

Main finding and its relevance. Pretraining on a brain gives a large advantage on that brain. Under SpatialNCE, spreading a fixed patch budget over more brains gave no detectable improvement on unseen brains. A larger pool of distinct patches from the same brains did help. For brain mapping, scanned tissue is therefore worth sampling more completely. A brain to be mapped should be included in pretraining, which needs no area labels from it. The same question arises wherever many observations come from few costly sources. In Earth observation, for example, image tiles are nested in acquired scenes and located in shared geographic coordinates. More data can then mean more tiles from existing scenes, more scenes, or broader geographic coverage. Our result shows that large differences between sources do not by themselves justify spreading a fixed budget over more sources. The matched comparison offers a way to test this in each setting.

Planned revisions. We will make the theoretical scope and the practical implications of our experiment more explicit in Sections 1 and 5. We will restrict the claims to the tested SpatialNCE setting. We will also clarify the probe protocol, the registration, and the presentation of the results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.