acceptodds
Withdrawn from ICLR 2027

The Final Layer Is Rarely Best: Layer-Wise Probing of Visual Foundation Models for 3D Reconstruction and Semantics

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual foundation models (VFMs) are routinely reused as frozen encoders, yet features are often taken from the final layer without testing whether it is appropriate for a dense downstream task. We study layer selection across 14 VFMs and five probes spanning static geometry, texture, joint Gaussian reconstruction, dynamic reconstruction, and semantic segmentation. Our evaluation uses a common Gaussian representation: static novel-view synthesis follows Feat2GS, while Feat4DGS adds a deformation readout for dynamic scenes and dedicated linear probes for semantics. Across the resulting 70 model–task comparisons, a non-final layer strictly outperforms the final layer in 64 cases. The preferred depth is nevertheless task- and model-dependent: the earliest sampled layer is best for texture in 10 of 14 models, whereas semantic and dynamic optima are distributed across the network. No pretraining objective dominates every probe; the best geometry, texture, joint reconstruction, dynamic reconstruction, and semantic scores come from five different model families. Separate deformation readouts also improve mean dynamic-scene PSNR by 4.26 dB over a shared MLP. These results show that layer choice is a first-order experimental variable and provide task-specific guidance for using frozen VFMs in dense 3D and semantic applications.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
novel view synthesis, gaussian splatting, visual foundation models, layerwise probing, semantic segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_wqeyOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper builds upon the static 3DGS probing framework of Feat2GS (CVPR 2025) and extends it to dynamic scenes (Deformation mode) and semantic segmentation (Semantic mode), collectively termed Feat4DGS. Using this setup, the authors evaluate 14 frozen VFMs across 5 layer depths (P10, P50, P80, Penultimate, Final) and 5 dense tasks, concluding that a non-final layer strictly outperforms the final layer in 64 out of 70 comparisons and that optimal depth is model- and task-dependent.

Critical strengths
  1. Standardized Large-Scale Evaluation: The paper standardizes model capacity (~300M ViT-L checkpoints) and feature dimensionality across 14 diverse VFMs, providing a thorough layer-wise evaluation that was missing in the original Feat2GS paper.
  2. Objective Qualitative Selection Protocol: The median-based automated view selection protocol in Appendix A.3 is well-designed and avoids subjective cherry-picking.
Critical weaknesses
  1. Limited Fidelity and Signal-to-Noise Ratio in Dynamic and Semantic Extensions: Compared to the clean static probes (G, T, A) inherited from Feat2GS, the two newly introduced extensions appear to face representation bottlenecks and optimization noise. In Dynamic mode, the best PSNR across all 14 VFMs falls within a narrow, low-fidelity band of 16.96--17.51 dB and $\sim$0.50 SSIM (range of 0.015), with noticeable visual blurring and tearing (Figs. 2a,). Because image VFMs lack temporal priors, the shallow MLP may primarily be fitting the timestamp tt under noisy Easi3R poses. Moreover, optimal layers often shift across metrics (PSNR vs. LPIPS, Table 6), suggesting that small layer-wise variations ($\sim$0.03--0.2 dB) might reflect optimization noise rather than motion representation quality. In semantic mode, projecting features onto 3DGS introduces coarse alpha-blending artifacts and boundary bleeding (Figs. 2b). Quantitatively, the layer-wise sensitivity is quite modest: the median gain over Final is only +0.45% mIoU, with 10/14 models showing ≤1.0%\le 1.0\% variation (Table 3).

  2. Scope and Standalone Contribution Relative to Feat2GS: Because the dynamic and semantic probes exhibit relatively low signal-to-noise ratios, the most compelling empirical findings still stem from the unchanged static G/T/A pipeline on the same 6 scenes from Feat2GS. Thus, the core validated contribution functions largely as a layer-wise extension of Feat2GS. While highlighting layer-selection sensitivity in Feat2GS is a useful observation, the effective methodological expansion beyond Feat2GS feels somewhat incremental for a standalone paper.

  3. Potential Confounding Factors in the Layer-Wise Evaluation: First, the "64/70" statistic compares the maximum over 4 candidate layers on test views against 1 fixed layer (Final) without a validation split. As shown in Tables 5 and 6, a model's optimal layer varies considerably across individual scenes and metrics; without a single intermediate layer that generalizes consistently across unseen scenes, post-hoc selection may introduce a positive bias. Second, compressing all layers to 256-dim via PCA and using a 2-layer MLP makes it difficult to disentangle intrinsic feature quality from probe capacity, as fixed-rank PCA and a shallow MLP may struggle to unpack the non-linear transformations in deeper ViT layers.

Questions for authors
  1. If the intermediate layer for each model is selected via leave-one-scene-out cross-validation (rather than post-hoc test selection) and evaluated with a modest tolerance threshold (e.g., ΔPSNR≥0.15 dB\Delta\text{PSNR} \ge 0.15\text{ dB}, ΔmIoU≥1.0%\Delta\text{mIoU} \ge 1.0\%), in how many of the 70 cases does the chosen non-final layer still outperform Final?

  2. Given the $\sim$17 dB PSNR baseline in Dynamic mode and the 0.45% median mIoU gap in Semantic mode, how can we best distinguish genuine representation differences from optimization variance? Also, how does the 3DGS semantic probe compare to direct 2D pixel-wise linear probing on the same views?

  3. What is the explained variance ratio retained by the 256-dim PCA at P10 vs. Final? How does a multi-layer fusion baseline (P10+P50+P80+Final followed by PCA) perform relative to the best single layer and the Final layer?

Reviewer_XtgNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents an empirical study of which layer of a frozen pretrained visual/geometry foundation model should be used for 3D and semantic tasks. A thorough comparison across 14 encoders (including ViT, VGGT, DINO, etc) is performed at different depth levels to evaluate the downstream task accuracy. Experiments cover indoor, static, and dynamic scenes. The main finding is that a non-final layer usually performs better than the final layer in the comparisons, though no single depth is a universally best choice. Also, layer selection can change which encoder performs best. An ablation study shows that separate deformation readouts improve PSNR over a shared readout layer.

Critical strengths
  1. The chosen topic is an interesting one that covers the most recent visual foundation backbones. The thorough study across 14 different backbones and several different layer choices is an academically rigorous setup.

  2. The result, though not fully surprising, is a true reflection and honest reporting of the performance. For practionioners that try to utilize these visual foundation features, this is a good guidance.

Critical weaknesses
  1. I found no clear results nor novel intuition gained from reading this paper. Comparing the results using different backbones and different layers, all of them are quite bad. With an average PSNR of <20 one can hardly tell whether the method "really" works or not. This is also justified by watching the qualitative results -- most of the rendered images are quite blurry and does not have any details. The same case applies for semantics mIoU, and the metrics vary very little across different setups. Each of the setup is performed only once, and it is hard to tell whether the differences are due to real feature differences or pure random noise.

  2. The experiment scale is too small to make the conclusion general enough. The results are only tested on <20 scenes, and this is simply not enough to reflect the generalization of the method. Moreover, one important method of fusing multiple layers is not explored at all.

  3. The method has limited novelty. It is a simple extension of the Feat2GS method, while adding only the dynamic probing part on top. Similar for the semantic readout heads. The setup is too simple to reach a confident conclusion.

Questions for authors
  • Could you report run-to-run variance (e.g., 3 seeds) for the setups to reflect signal-to-noise ratio?

  • Is PCA fit per scene, per model-layer, or globally?

  • In the dynamic setting, camera poses are jointly optimized with the readout. Are the final optimized poses different across layers of the same model? If so, how much of the PSNR variation across depth is attributable to pose convergence rather than representation quality?

Reviewer_YQmyOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Extending Feat2GS, the author proposed to analyze the effect on the downstream task (particularly 3D related) given different VFMs with different training strategies and optimization targets. The study is on how/what is the effect of the layer choice in the downstream task performance. The authors tested various different VFMs with different optimization targets and different downstream tasks and found that the performance do vary slightly from layer to layer.

Critical strengths

The work studied how different VFMs can perform in the 3D reconstruction related downstream task. It covers a very wide range of VFMs (for both 2D or 3D for their original task) and different layers. This creates a broad suite of empirical data points from which different conclusions about layer and/or model/optimization target can be drawn from.

The work reveals some interesting aspects that is mostly unavailable in existing works, such as 3D VFM features doesn’t necessarily perform the best in downstream 3D related tasks.

Critical weaknesses

The work doesn’t further dive into the different components of why the layer matter. It for the most part only provided some data points on which layer (often not the last) is the one most suitable (by a small margin).

(note that the fact that the difference is small by itself is only an empirical observation, it is what it is, that is not a weakness in itself in my opinion. The weakness is in the lack of further proper interpretation and study)

Usually I wouldn’t regard statistical significance an important component of the scientific knowledge discovery, but I think they are important for this work and is mostly missing here. First, the differences are very small, it means the conclusion presented without multiple different runs (and ideally initializations) of the VFMs are nearly noise, or at least we could not tell. Second, there are a lot of different models and layers selected, just by pure chance, the last layer is likely not the best layer among many layers anyways. And whether that is true or not, requires additional statistical measurement to nail down.

Unfortunately, I find as the paper stands in its current form, has limited novelty. And its key empirical data points: "which layer is the best and by how much" is not very meaningful without proper statistical measurement.

Questions for authors

How much of the observed difference is due to the pretraining objective, versus architecture, dataset, and optimization; i.e. the original objective for training the VFM.

How much does an intermediate representation's advantage reflect actual geometric information, versus spatial detail, feature consistency. Or is the advantage as a result of ease of fitting the Gaussian probe - i.e. the information were available either way, but the intermediate layer feature is usable.

Any statistical significance measurement regarding at least some of the reports? Is which layer is the best a stable thing for a model - downstream task pair, or it is rather stochastic?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.