acceptodds
Under review as a conference paper at ICLR 2027

Reduce, Then Encode: Multiscale Volumetric Reduction for 2D Foundation Models in Brain MRI

Abstract

Pretrained 2D foundation models offer a practical alternative to dedicated 3D pretraining for brain structural magnetic resonance imaging (sMRI), but their use on volumetric data requires bridging the mismatch between a 2D encoder and a 3D volume input. Existing methods typically encode slices independently and integrate their features afterwards. We introduce Multiscale Volumetric Reduction (MVR), a reduce-then-encode approach that compresses each anatomical view from \(D\) slices into \(M << D\) complementary 2D components before foundation-model encoding. MVR combines an uncentered-PCA base component derived from the original through-plane intensities with residual detail components constructed from multiscale spatial descriptors. The reduction is estimated from the training volumes without diagnostic labels or gradient-based optimization and remains fixed thereafter. The resulting components are independently processed by a shared frozen 2D foundation model and concatenated for linear probing. Under this frozen-encoder setting, MVR achieves strong overall performance across ADNI, OASIS, and ABIDE relative to the evaluated 2D-to-3D adaptation methods and simple input-reduction baselines, while also generalizing strongly from ADNI to AIBL.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
pretrained 2D foundation models, 3D medical volumes, volumetric reduction, medical image classification

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_fPEjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MVR, a method for using 2D foundation models with 3D brain MRI data. Instead of processing MRI slices separately, the method first reduces the 3D volume into a smaller set of 2D images using PCA and multiscale information. These images are then passed through a frozen DINOv3 model.

The authors evaluate the method on ADNI, OASIS, and ABIDE, with an additional test on AIBL. The results are generally better than the selected baselines, especially when the foundation model is frozen.

Critical strengths
  1. The main idea is simple and practical. Reducing the MRI volume before passing it to a 2D foundation model is an interesting alternative to processing individual slices. The method also does not require additional training of the foundation model.

  2. The experiments cover multiple datasets and include external validation on AIBL. I also like the ablation studies, which show that both the multiscale information and the detail components help improve performance.

Critical weaknesses
  1. My main concern is that the method is tested with only one 2D foundation model, DINOv3. Since the paper proposes a general approach for 2D foundation models, it is not clear whether the same improvements would be seen with other models. Also, most parts of the method are based on existing techniques such as PCA and Gaussian smoothing, so the methodological novelty seems somewhat limited.

  2. Some improvements over the baselines are quite small. For example, on ABIDE, MVR achieves an AUC of 66.02, compared to 65.99 for uniform slice sampling. The paper does not provide statistical tests to show whether these differences are significant. The computational cost is also not fully evaluated, although efficiency is one of the motivations for the method.

Questions for authors
  1. Have the authors tested MVR with other 2D foundation models, such as DINOv2 or a medical imaging model? I think this would help show whether the method works well beyond DINOv3.

  2. Can the authors provide statistical significance tests for the main comparisons? Also, how much inference time and GPU memory does MVR require compared to the other methods? This would help better understand the practical benefits of reducing the volume before encoding.

4 comments

We thank the reviewer for the constructive feedback.

Generalization Across 2D Foundation Models

As suggested, we additionally evaluated MVR using a frozen DINOv2 encoder on ADNI. The results are presented below. Our MVR achieves the highest AUC and BAcc, outperforming strong second-best RAPTOR, AnyMC3D, and Eigenslices. In particular, MVR exceeds Eigenslices by 3.51% AUC and 2.86% BAcc under matched encoder and downstream classification settings.

MethodAUC (%)BAcc (%)
RAPTOR80.26 ± 0.9472.91 ± 2.18
AnyMC3D84.24 ± 3.0274.81 ± 1.94
Eigenslices85.50 ± 3.2678.62 ± 1.43
MVR (DINOv2)89.01 ± 1.6981.48 ± 1.43

Novelty of MVR

Our MVR is a carefully designed method rather than a trivial combination of existing techniques.

1. Novel volumetric reduction design. Our volumetric reduction goes beyond conventional PCA-based methods such as Eigenslices. Specifically, we first construct descriptors that combine through-plane information with in-plane spatial variation at multiscales, providing a diverse information basis for reduction. Second, we design a base-and-detail construction scheme that separates a dominant intensity-based reference from complementary residual multiscale variation, enabling the detail components to capture variation not linearly explained by the base reference. These designs create a new representation space in which PCA is performed and allow complementary information to be allocated across the reduced components. Its superiority has been demonstrated by the favorable performance improvements across ADNI, OASIS, ABIDE and AIBL datasets shown in Tables 1 and 2.

2. New architecture level design. Existing approaches that utilize forzen 2D foundation models, such as RAPTOR and AnyMC3D, primarily incorporate cross-slice information after independently encoding each slice using 2D foundation models, which may lose structure infomation across slices. Our MVR instead constructs volume-informed inputs before sending to 2D foundation models, preseving information across slices. The experimental results in both the main paper and the rebutall demonstrate this is a promising architecture worthing further exploration in the future.

Our additional experiments below on AdrenalMNIST3D (abdominal CT) and VesselMNIST3D (brain MRA) (Yang et al., 2023) further demonstrate the superiority of MVR for adapting frozen 2D foundation models to medical volumes beyond brain MRI.

MethodAdrenal AUC (%)Adrenal Accuracy (%)Vessel AUC (%)Vessel Accuracy (%)
RAPTOR (DINOv3)90.9986.2494.6492.15
AnyMC3D91.5087.2586.1890.84
Eigenslices87.8683.8994.1893.09
MVR93.2087.2596.6293.72

Statistical Uncertainty

As suggested, we conducted a paired, class-stratified bootstrap analysis with 5,000 iterations to measure two-sided 95% confidence intervals for AUC differences, similar to RAPTOR (An et al., 2025).

Compared to second-best methods The paired comparisons with second-best methods below provide additional statistical evidence for MVR’s AUC improvements. It shows the performance superiority is statistically significant in 6 of 7 comparisons based on 95% CIs across four datasets. Specifically, on ADNI, OASIS, and the out-of-distribution AIBL evaluation, MVR achieves positive AUC differences over the evaluated competitive methods, with all differences being statistically significant. On ABIDE, the difference relative to uniform slice sampling remains inconclusive. However, MVR achieves substantially higher mean AUC than uniform slice sampling on ADNI, OASIS, and AIBL, with improvements of 7.98%, 8.15%, and 9.32%. Overall, these results support the generally favorable performance of our method.

DatasetMVR vs. BaselineΔAUCAUC 95% CI
ADNIMVR vs. MedicalNet+2.25[0.33, 4.29]
ADNIMVR vs. 3DINO+2.89[0.92, 4.96]
OASISMVR vs. MedicalNet+1.95[0.15, 3.58]
OASISMVR vs. Eigenslices+1.37[0.02, 2.83]
AIBLMVR vs. MedicalNet+1.06[0.03, 2.21]
AIBLMVR vs. 3DINO+3.54[0.12, 7.04]
ABIDEMVR vs. Uniform slice sampling+0.03[−1.56, 1.54]

Statical significance in Ablation study As shown below, removing the "detail components" consistently reduces AUC across all three datasets, with all three CIs positive difference. Removing "Gaussian scale space" also reduces AUC across datasets, with two of three CIs showing statisticslly significant improvement. Overall, These results provide additional statistical support for the contributions of the proposed multiscale construction and residual detail components.

DatasetMVR vs. Ablation VariantΔAUCAUC 95% CI
ADNIMVR vs. w/o Gaussian scale space+1.84[0.41, 3.11]
ADNIMVR vs. w/o detail components+3.41[1.72, 5.13]
OASISMVR vs. w/o Gaussian scale space+0.91[−0.36, 2.21]
OASISMVR vs. w/o detail components+3.01[0.95, 5.15]
ABIDEMVR vs. w/o Gaussian scale space+2.64[0.76, 4.90]
ABIDEMVR vs. w/o detail components+4.49[2.02, 6.83]

Inference Efficiency

We measured inference efficiency on ADNI using the same NVIDIA H200 GPU for compared methods.

Our MVR achieves the highest inference speed while achieving best AUC. Our MVR is approximately twice as fast as Eigenslices, 54 times as fast as RAPTOR, and 1.4 times as fast as AnyMC3D. Although MVR requires slightly higher peak GPU memory, it is affordable even with a consumer PC.

MethodVolumes/sPeak VRAM (GB)AUC (%)BAcc (%)
Eigenslices3.111.1984.97 ± 2.2776.89 ± 3.43
RAPTOR (DINOv3)0.111.1984.93 ± 1.9677.62 ± 4.32
AnyMC3D4.391.29584.45 ± 2.9376.90 ± 3.81
MVR6.142.7088.85 ± 1.2280.92 ± 1.02
Reviewer_fPEjOpenReview ↗

Score Update After Author Rebuttal

The authors' responses addressed some of my concerns, but my main concern about the novelty of the basis and details remains. Nevertheless, I think this is a good paper, and I have raised my score to 3.

We thank the reviewer for their constructive feedback and for reconsidering the assessment. We appreciate the positive evaluation of our work. We will further clarify the technical contributions and incorporate the additional experimental evidence into the revised manuscript.

Reviewer_H4mHOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

For 3D brain MRI classification, existing approaches mainly either train models on the full 3D volumes, or use a 2D model to feed individual slices into and aggregate the features afterwards. The latter, while not requiring heavy training and much more compute- and data-efficient, still requires running inference over many 2D slices, which can be expensive, and since the 2D encoder is frozen, does not naturally preserve the volumetric structure (only post-hoc or with additional assumptions). The authors propose an alternative approach, where they combine the information across slices beforehand and then pass those into the 2D model. Information across slices is combined via one base image and fifteen detail images (per view direction) which are constructed from differences between progressively blurred slices (PCA is used to maximize orthogonal directions). A frozen 2D model then encodes these images from a single volume into a volumetric embedding. Downstream classification is done with a logistic regression trained on the volume embeddings. Across three datasets, they generally observe better performance than other baselines, and ablations support the modeling choices.

Critical strengths
  1. The paper is clear and well-written. The method is specified properly via well-defined equations, and the figures are informative. Preprocessing and splits seem to follow best practices and are consistently applied across methods. The authors also look at three different datasets which cover two diseases and evaluate several other baselines.

  2. The ablations are comprehensive and seem to directionally support their modeling choices (though see weakness 1). I also found the addition of the AIBL dataset as an external validation of the ADNI-trained model to be a nice touch/good experiment to show, since this helps validate that the learned features generalize to another independent cohort instead of just showing ADNI-specific characteristics.

Critical weaknesses
  1. My main major concern is regarding the reports of uncertainty throughout all comparisons in the paper. I notice that only fold SDs are shown, instead of 95% confidence intervals (or paired tests). When looking at the strongest baselines and most ablations, despite the point estimates being marginally higher, the intervals between methods already overlap quite a bit, and I would expect them to become even larger if 95% CIs are reported instead. More specifically, for the AIBL validation, it looks like the ADNI models are all evaluated on the exact same test set, so in this case, the intervals don’t even represent any of the test set variation. Without proper CIs (ideally non-overlapping), I can’t be convinced that the differences are significant or the claims for being the best method hold.

  2. The paper makes a case for improved performance when the encoder is frozen. They state that they do this to “isolate how volumetric information is presented to a pretrained 2D representation”. However, I’m still unclear on why this would be an important realistic scenario. For example, are there any efficiency claims to be made? MVR also performs worse than AnyMC3D when both are fine-tuned with LoRA on ADNI. Why is that an acceptable outcome here? I believe the authors should elaborate more on the frozen use case.

Questions for authors
  1. Raptor can be run with DINOv3, as its encoder is not fixed to DINOv2; how does it perform when it's run with DINOv3 instead?
  2. My impression is that the performance of this method heavily depends on the quality of the reduction step, which would depend on the train/test split. How robust is the method to limited training sample? Given 3D tasks generally have small available data, a practical method should be able to reliably perform reduction.
4 comments

We thank the reviewer for the constructive feedback.

Paired Bootstrap Analysis

As suggested, we conducted a paired, class-stratified bootstrap analysis with 5,000 iterations to measure two-sided 95% confidence intervals for AUC differences, similar to RAPTOR (An et al., 2025).

Compared to second-best methods

The paired comparisons with second-best methods below provide additional statistical evidence for MVR’s AUC improvements. It shows the performance superiority is statistically significant in 6 of 7 comparisons based on 95% CIs across four datasets. Specifically, on ADNI, OASIS, and the out-of-distribution AIBL evaluation, MVR achieves positive AUC differences over the evaluated competitive methods, with all differences being statistically significant. On ABIDE, the difference relative to uniform slice sampling remains inconclusive. However, MVR achieves substantially higher mean AUC than uniform slice sampling on ADNI, OASIS, and AIBL, with improvements of 7.98%, 8.15%, and 9.32%. Overall, these results support the generally favorable performance of our method.

DatasetMVR vs. BaselineΔAUCAUC 95% CI
ADNIMVR vs. MedicalNet+2.25[0.33, 4.29]
ADNIMVR vs. 3DINO+2.89[0.92, 4.96]
OASISMVR vs. MedicalNet+1.95[0.15, 3.58]
OASISMVR vs. Eigenslices+1.37[0.02, 2.83]
AIBLMVR vs. MedicalNet+1.06[0.03, 2.21]
AIBLMVR vs. 3DINO+3.54[0.12, 7.04]
ABIDEMVR vs. Uniform slice sampling+0.03[−1.56, 1.54]

Statical significance in Ablation study

As shown below, removing the "detail components" consistently reduces AUC across all three datasets, with all three CIs positive difference. Removing "Gaussian scale space" also reduces AUC across datasets, with two of three CIs showing statisticslly significant improvement. Overall, These results provide additional statistical support for the contributions of the proposed multiscale construction and residual detail components.

DatasetMVR vs. Ablation VariantΔAUCAUC 95% CI
ADNIMVR vs. w/o Gaussian scale space+1.84[0.41, 3.11]
ADNIMVR vs. w/o detail components+3.41[1.72, 5.13]
OASISMVR vs. w/o Gaussian scale space+0.91[−0.36, 2.21]
OASISMVR vs. w/o detail components+3.01[0.95, 5.15]
ABIDEMVR vs. w/o Gaussian scale space+2.64[0.76, 4.90]
ABIDEMVR vs. w/o detail components+4.49[2.02, 6.83]

Realistic Scenarios for Frozen Foundation Models

Frozen foundation models provide practical advantages for medical imaging applications, particularly when labelled data and computational resources are limited. Below we list several representative benefits and existing works that advocate the use of 2D foundation models.

  • Reduced training cost. Keeping the foundation model frozen eliminates the need for encoder backpropagation and task-specific fine-tuning. This substantially simplifies downstream adaptation, allowing pretrained representations to be used without repeatedly optimising a large foundation model.

  • Limited labelled data. When labelled medical data are limited, fine-tuning a foundation model can overfit to a small training cohort. Freezing the encoder preserves its pretrained representations and limits supervised optimisation to a lightweight classifier, reducing the risk of overfitting to task-specific training data.

  • Reuse of pretrained knowledge. General purpose 2D foundation models contain representations learned from large-scale natural images. Keeping the pretrained encoder frozen allows these representations to be reused for volumetric medical imaging without additional domain specific encoder training.

  • Existing work that advocate the use of frozen foundation models. These advantages have motivated several studies in medical imaging. RAPTOR (An et al., 2025) highlights the substantial data and computational requirements of dedicated 3D medical model training and demonstrates frozen 2D foundation models as a practical alternative. DinoAtten3D (Rafsani et al., 2025) leverages frozen DINOv2 representations to address medical classification with limited annotations, while DINO-Reg (Song et al., 2025) demonstrates their applicability to multimodal medical image registration without encoder fine-tuning. These works support the practical relevance of frozen foundation-model reuse, which MVR further investigates through input-space volumetric reduction.

Inference Efficiency

We measured inference efficiency on ADNI using the same NVIDIA H200 GPU for compared methods.

Our MVR achieves the highest inference speed while achieving best AUC. Our MVR is approximately twice as fast as Eigenslices, 54 times as fast as RAPTOR, and 1.4 times as fast as AnyMC3D. Although MVR requires slightly higher peak GPU memory, it is affordable even with a consumer PC.

MethodVolumes/sPeak VRAM (GB)AUC (%)BAcc (%)
Eigenslices3.111.1984.97 ± 2.2776.89 ± 3.43
RAPTOR (DINOv3)0.111.1984.93 ± 1.9677.62 ± 4.32
AnyMC3D4.391.29584.45 ± 2.9376.90 ± 3.81
MVR6.142.7088.85 ± 1.2280.92 ± 1.02

Performance under LoRA Adaptation

AnyMC3D is designed to jointly optimise its LoRA-adapted foundation model and query model, whereas MVR uses a fixed reducer. End-to-end optimisation with learnable aggregation usually leads to better performance. However, when only a small number of training data is avaiable, such as dataset AIBL (143–144 training subjects), our MVR achieves better performance than AnyMC3D as shown in the table below.

MethodAUC (%)BAcc (%)
AnyMC3D + LoRA84.54 ± 8.2169.93 ± 15.70
MVR + LoRA86.03 ± 5.8777.08 ± 5.61

RAPTOR with DINOv3

As suggested, we additionally evaluated RAPTOR with a frozen DINOv3 encoder.

As shown below, our MVR achieves highest AUC and BAcc across all four cohorts. On ADNI, MVR improves AUC and BAcc by 3.92% and 3.30%. On AIBL, the corresponding improvements are 5.54% and 6.38%. These results show that our MVR consistently performs better than the second best methods.

DatasetAUC (%)BAcc (%)Δ\DeltaAUCΔ\DeltaBAcc
ADNI84.93 ± 1.9677.62 ± 4.32+3.92+3.30
OASIS68.64 ± 3.0262.72 ± 2.55+1.66+1.77
ABIDE58.91 ± 1.0356.11 ± 1.94+7.11+4.92
AIBL81.73 ± 1.5566.09 ± 2.09+5.54+6.38

Differences (Δ) are calculated as MVR minus RAPTOR, in percentage points.

Robustness to Limited MVR Fitting Data

To evaluate the robustness of MVR under limited fitting data, we varied the fraction of training subjects used to estimate the reduction operators from 10% to 100% on ADNI, while keeping the downstream classifier-training protocol unchanged. MVR maintains relatively stable performance across these settings. Notably, using only 10% of the training subjects (82 per fold), MVR achieves 88.43% AUC and 80.73% BAcc, compared with 88.85% and 80.92% using the full training set. These results demonstrate that MVR's reduction operators can be estimated from relatively small training subsets without substantial degradation in the observed performance.

MVR fitting dataSubjects per foldAUC (%)BAcc (%)
10%8288.43 ± 2.0680.73 ± 1.60
25%20588.05 ± 1.5383.04 ± 1.67
50%410–41188.26 ± 1.4581.06 ± 1.92
100%821–82288.85 ± 1.2280.92 ± 1.02

I thank the authors for their detailed response.

I think the paper is technically sound and the results are robust. I recommend that some of the new results are added to the camera-ready version to further highlight the contributions.

As most of my concerns were adequately addressed, I have raised my score to 3.

We thank the reviewer for their thoughtful feedback and for reconsidering the mark after our rebuttal. We are glad that the additional experiments and analyses addressed most of the concerns. As suggested, we will incorporate the new results into the revised manuscript to further strengthen the empirical evidence and presentation.

Reviewer_AW76OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This manuscript propose a generic mechanism to learn 3D features of medical images using frozen 2D foundation models , allowing for better generalization. The mechanism involves reducing the 2D across main axis using PCA and then performing multi-scale difference of gaussians PCA without the main template , emphasizing the essential distinctive features to retain when running the frozen 2D model instead of focussing on the “mean” looking brain. This leads to improved learning on 3D brain MRI volumes compared to standard slicing and aggregation technique followed before.

Critical strengths
  • The problem of adapting 2D foundations to 3D by SSL on 3D data is actually one of the most pressing challenges in the medical image domain in general.

  • The paper is easy to read and follow, the presentation quality is also great and facilitator to the reader understanding the content of the work ( see Figure 1 , 2 and 3 for examples).

  • The evaluations are fair and standard baselines comparisons are present.

  • The method is simple and can be extended to any 3D medical image problem.

Critical weaknesses
  • The empirical results in the manuscript are not sufficient to demonstrate the capability beyond the three brain MRI datasets. The promise of the paper is big and in theory it can be applied to more than just brain MRI and simple classification.

  • No inclusion of the code to reproduce the work puts a limitation on the usability of the work.

  • Some important ablations are missing on the fronzen encoder and some other aspects of the pipeline.

Questions for authors

can the method be extended to other problems in 3D medical imaging ? Could you demonstrate this ?

3 comments

We thank the reviewer for the constructive feedback.

Generalization Beyond Brain MRI

To evaluate MVR beyond brain sMRI diagnosis, we conducted additional experiments on AdrenalMNIST3D and VesselMNIST3D (Yang et al., 2023). AdrenalMNIST3D is a CT dataset for hyperplasia classification, while VesselMNIST3D is for aneurysms classification.

Our MVR achieves the highest AUC on both datasets and the highest accuracy on VesselMNIST3D, while achieving the joint-highest accuracy on AdrenalMNIST3D. Compared with RAPTOR, MVR improves AUC by 2.21% and 1.98% and accuracy by 1.01% and 1.57% on AdrenalMNIST3D and VesselMNIST3D. These results demonstrate the potential of MVR extending its application beyond brain MRI.

MethodAdrenal AUC (%)Adrenal Accuracy (%)Vessel AUC (%)Vessel Accuracy (%)
RAPTOR (DINOv3)90.9986.2494.6492.15
AnyMC3D91.5087.2586.1890.84
Eigenslices87.8683.8994.1893.09
MVR93.2087.2596.6293.72

Code Availability

We have uploaded a pre-release implementation of the core component of MVR, reduction-fitting pipeline, in an anonymous repository:

https://anonymous.4open.science/r/MVR-A81D/

The full codebase will be released to public depending on the acceptance of this work.

Additional Evaluation with a Frozen Encoder

As suggested, we additionally evaluated MVR using a frozen DINOv2 encoder on ADNI. The results are presented below. Our MVR achieves the highest AUC and BAcc, outperforming strong second-best RAPTOR, AnyMC3D, and Eigenslices. In particular, MVR exceeds Eigenslices by 3.51% AUC and 2.86% BAcc under matched encoder and downstream classification settings.

MethodAUC (%)BAcc (%)
RAPTOR80.26 ± 0.9472.91 ± 2.18
AnyMC3D84.24 ± 3.0274.81 ± 1.94
Eigenslices85.50 ± 3.2678.62 ± 1.43
MVR (DINOv2)89.01 ± 1.6981.48 ± 1.43

thanks for the reply, it addressed my main concerns of the paper.

We thank the reviewer for their positive feedback and for acknowledging that our additional results addressed the main concerns. We appreciate the constructive suggestions and will incorporate the new experimental results and clarifications into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.