acceptodds
Under review as a conference paper at ICLR 2027

SALF: Spatially Attentive Layer Fusion for Probing Vision Transformers

Abstract

Adapting a pre-trained Vision Transformer (ViT) to a downstream task requires extracting a single compact representation from multi-layer patch tokens. Current methods either attentively probe the patches of the final layer (discarding intermediate features) or fuse per-layer average-pooled summaries across depth (discarding fine-grained spatial detail). By probing all patches across all layers on multiple datasets, we demonstrate that the final layer is rarely the most informative and that dynamic spatial aggregation significantly outperforms static average pooling. Notably, our analysis reveals that this spatial aggregation plays a more dominant role in downstream performance than depth-wise fusion, effectively reducing the reliance on complex multi-layer integration. To efficiently leverage both dimensions, we factorize all-token readout into SALF-P (Spatially Attentive Layer Fusion with Per-layer aggregation), which first aggregates the spatial patches of each layer using a parameter-efficient shared cross-attention module, followed by a second cross-attention module to fuse these features across layers. However, further analysis reveals that depth-wise fusion has limited impact when paired with spatial attention. This motivates our SALF-A, which averages over layer features before applying a single attentive pooling. SALF-A matches SALF-P across our benchmarks, while reducing the spatial readout to one pass and shrinking the token cache by a factor of the network depth. Evaluations across 19 downstream datasets and seven frozen backbones, alongside Image Quality Assessment (IQA) tasks under parameter-efficient fine-tuning, confirm that SALF-P and SALF-A substantially improve accuracy on fine-grained visual perception tasks by up to 16.20 percentage points over ALF while maintaining strong performance on globally semantic tasks.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Representation Learning, Attentive Probing, Vision Transformers, Intermediate Layer Fusion, Transfer Learning, Parameter Efficiency

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_BP1xOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates spatial aggregation and cross-layer fusion for probing pretrained Vision Transformers. The authors propose SALF-P, which combines per-layer spatial attention with layer fusion, and SALF-A, which averages features across layers before applying a single spatial attention module. The central claim is that effective spatial aggregation reduces the need for sophisticated layer fusion. Experiments across multiple classification benchmarks, pretrained backbones, and image quality assessment tasks demonstrate promising results, particularly on spatially sensitive tasks.

Critical strengths
  1. The motivation is clear. This paper presents a logical progression from empirical observations to method design, with SALF-A offering a simple and efficient alternative.
  2. The evaluations cover diverse 19 datasets and backbone families, with detailed ablations and efficiency analyses.
  3. The investigations into spatial versus layer-wise information, including averaging efficiency and spatial stationarity, provide potentially useful insights.
Critical weaknesses
  1. Limited novelty beyond ALF. The closely related ALF paper (Ciernik et al., ICML 2026) already investigates the complementarity of spatial and hierarchical aggregation (Sec. 4.6) and evaluates a hybrid probing strategy attending to spatial tokens across multiple layers (Appendix A.9). SALF primarily extends this direction through different aggregation orders and factorization. The conceptual advancement is therefore somewhat incremental, and a direct comparison with ALF's hybrid variant is missing.

  2. Despite the extensive evaluation, the improvements over ALF are overwhelmingly driven by only four datasets (Aircraft, GTSRB, SVHN, and DMLab). Excluding these datasets, the average gains across the remaining 14 benchmarks in Table 1 become nearly negligible: +0.23 pp for SALF-P and +0.02 pp for SALF-A. This substantially weakens the evidence for broad effectiveness. While the authors acknowledge task-dependent gains, the extent of this concentration is insufficiently emphasized.

  3. The paper repeatedly highlights evaluations on 19 classification benchmarks, yet the main Table 1 reports only 18. The remaining benchmark, ImageNet-1k, is presented separately in Supplementary Table 12, using only five backbones rather than seven. Notably, SALF-A slightly underperforms ALF on ImageNet-1k on average (81.05 vs. 81.27), while SALF-P achieves only a modest improvement (+0.76 pp). Although the experiments are reported, the rationale for separating this important benchmark and the precise scope of the headline aggregate results should be clarified.

  4. Several of the largest improvements occur on tasks exhibiting predominantly static spatial attention patterns. It remains unclear whether the gains genuinely require content-adaptive attention or could largely be explained by simpler learned, input-independent spatial weighting. A controlled static spatial pooling baseline would help establish the necessity of the proposed mechanism.

  5. The full fine-tuning baseline receives less extensive hyperparameter tuning and lacks data augmentation, weakening the corresponding comparisons. Evaluation on newer foundation vision backbones would also strengthen the practical relevance.

Questions for authors

Idea is good, but there are major concerns in terms of the experiments, especially that the main gain comes from 4/18 benchmarks. And if we remove these four benchmarks, there will be near zero improvement. Also, it will be much better if the authors can add static learned pooling experiment for comparison.

I still keep open for this paper. If authors can justify the contributions and the mechanisms well, I may raise my rating to 3.


After rebuttal

Although the proposed method is conceptually incremental over ALF, the additional experiments have substantially strengthened the empirical evidence, particularly regarding the effectiveness of dynamic spatial attention and the practical efficiency of SALF-A. The new comparisons and clarifications have adequately addressed my main concerns. I therefore have raised my rating from 2 to 3 (WA).

Reviewer_CDn9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

SALF combines (spatial) patch and (global) CLS tokens from a ViT into a single representation for downstream applications like image classification. The aggregation module and final linear classifier are trained jointly. The linear-probe baseline uses the final CLS token, while ATT applies attention pooling to the final layer’s tokens. ALF uses all layers, but first averages the patches within each layer, then attends over these averages and the CLS tokens.

The paper introduces FLAT, which provides an unrestricted reference baseline: one learned query attends to all patch and CLS tokens from every layer at once. It preserves access to all intermediate features without prior pooling, but requires processing a large token sequence.

SALF proposes two cheaper approaches:

  • SALF-P first applies spatial attention within each layer to produce a summary. The attention weights are shared across layers, while each layer has its own learned query. A second attention module combines these summaries and the CLS tokens into the final representation. Its main difference from ALF is replacing uniform spatial averaging with learned attention.

  • SALF-A reverses the order of aggregation. It first averages corresponding tokens across layers, including the CLS tokens, then applies spatial attention once. This removes learned layer fusion and repeated spatial-attention passes. Its main difference from ALF is which information gets averaged: ALF averages across spatial positions, whereas SALF-A averages across depth while preserving separate spatial positions.

Critical strengths

The paper is well written and provides useful empirical insight: spatial information from intermediate layers can be used to train a better classifier (vs. simply relying on CLS token).

SALF-P represents an efficient variant of SALF-P, at similar average accuracy on the main benchmarks with less computation and feature storage.

The evaluation covers many datasets and backbone families.

Ablations demonstrate the roles of spatial pooling and layer aggregation.

Critical weaknesses

The novelty of the method is limited: the method mainly combines existing attention pooling and intermediate-layer aggregation.

The application scope is narrow: the experiments focus on global classification and image-quality regression.

SALF-A does not consistently match SALF-P. Their average performance on the main benchmarks is similar but SALF-A has ~1% lower accuracy on ImageNet than SALF-P; ~4% lower on Country-211.

Questions for authors

What is the main novel aspect over ALF and VQT (beyond combining spatial attention and intermediate-layer aggregation)?

Did you test SALF on retrieval, dense prediction, or VLM integration?

How does SALF compare with a learned spatial weighting?

Can you explain when depth averaging loses useful information, especially given SALF-A’s lower accuracy relative to SALF-P on ImageNet and Country-211?

Reviewer_BypDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to extract a compact representation from the multi-layer patch tokens of a frozen ViT. Existing probes keep only part of the information: last-layer attentive probing (ATT) ignores intermediate layers, and Attentive Layer Fusion (ALF) average-pools the patches within each layer.

A diagnostic layer sweep shows that attentive spatial pooling clearly beats average pooling and that intermediate layers are often more informative than the last one. It also suggests that, once spatial attention is used, the form of cross-layer fusion matters little.

Building on this, the authors propose two probes:

  • SALF-P applies shared per-layer spatial cross-attention, then fuses the layers with a second cross-attention.
  • SALF-A averages tokens across layers, then applies a single attentive pooling, at roughly ATT’s cost.

Across 19 datasets and seven backbones, both variants raise the mean gain over the linear probe from about +5.6–5.9 pp (ALF, ATT) to about +8.6 pp. The gains are largest on spatially driven tasks such as SVHN, GTSRB and Aircraft. The paper also reports NR-IQA results and two statistics (η and ξ) that correlate with SALF’s advantage over ALF.

Critical strengths
  1. A simple multi-layer probe with broad, consistent evidence, at roughly last-layer-ATT cost. The evaluation covers 19 datasets, seven frozen backbones across three pre-training paradigms (plus MAE), ImageNet-1k, and NR-IQA under PEFT. The diagnostics in §3 use a re-partitioned training split, so they never touch the test data. Both variants raise the mean gain over the linear probe from +5.6 pp (ALF) and +5.9 pp (ATT) to about +8.6 pp (Table 1). They are also the best probes on every backbone (Fig. 3).

    SALF-A can cache the layer-mean tokens in advance. Its cache size (197 tokens), memory use (1.18 GiB) and inference cost (18.03 GFLOPs) therefore match ATT (Table 13). This is a practical option for multi-layer probing.

  2. Evidence that “intermediate features are needed” and “complex depth fusion is needed” are separate claims. Three ablations support the first claim without the second:

  • In Table 15, the four fusion schemes for SALF-A differ by at most 0.3 pp.

  • In Table 16, the width of the across-layer attention barely matters.

  • In Table 17, keeping 4–6 layers that span the full depth matches using all 12 layers, but dropping the early layers costs 5–6 pp on GTSRB.

    The FLAT intervention in Table 18 adds a narrower point: a single spatial map shared across layers preserves accuracy. That experiment keeps the per-layer attention mass, so it does not by itself show that uniform depth averaging is sufficient.

Critical weaknesses
  1. The source of the gains is not cleanly isolated. Three comparisons leave the attribution open.
  • ATT uses a different head configuration. ATT uses 8 × 192 heads and SALF-A uses 24 × 64 (Table 5). The total inner width and parameter count are the same. However, the number of heads sets how many independent spatial distributions the probe can learn. The headline ATT → SALF-A gap therefore mixes the effect of multi-layer input with the effect of head count.

  • There is no input-independent learned spatial pooling baseline. The paper attributes its gains to dynamic, image-dependent spatial selection. Yet the ξ analysis finds the largest gains on static-dominant tasks, where the attended positions are largely shared across images. A shared, learned per-position weighting on the layer-averaged tokens would separate three effects:

    • the cost of uniform averaging,
    • the benefit of any non-uniform spatial weighting,
    • the additional benefit of conditioning the weights on each image.
  • The VQT comparison is not fully matched. VQT is trained without feature noise and with a different batch size. The classification results also appear to come from single runs. The reported ± values are taken across backbones, not across seeds.

  1. The gains are concentrated, and several claims need qualification. Based on the Table 1 means, excluding Aircraft, GTSRB and SVHN reduces SALF-A’s average advantage over ALF from +2.9 to +0.7 pp, and both SALF variants are approximately tied with VQT on the remaining datasets. These gains are valuable, but they indicate particular strength on spatially driven tasks rather than uniform superiority. Accordingly:
  • Global semantics: Both variants trail ALF on Country-211 (+1.84 and −2.21 vs. +3.52 pp), and SALF-A also trails ALF on the ImageNet mean, although SALF-P achieves the highest ImageNet mean. The abstract’s claim of strong performance on globally semantic tasks should be narrowed.
  • Full finetuning: The finetuning baseline uses fixed hyperparameters without the search applied to the probes, and no augmentation. It underperforms linear probing on 9 of 17 datasets. Claims of matching or exceeding finetuning should be limited to this configuration.
  • ξ analysis: ξ requires trained SALF-P attention maps, so the reported correlation supports post-hoc interpretation rather than an a-priori diagnostic.
  • Fusion ablations: All variants reuse the default model’s hyperparameters, and per-dataset differences can be larger than the mean suggests (e.g., 5.6 pp on DMLab in Table 15). The finding that complex fusion adds little therefore holds for the tested configuration.
Questions for authors
  1. What accuracy does SALF-A achieve when the layer-averaged tokens are replaced with last-layer tokens, keeping its architecture and training protocol unchanged? If Fig. 2 already includes this control, please confirm the head count and per-head dimension used.

  2. How does input-independent learned spatial pooling over the same layer-averaged tokens compare with SALF-A, particularly on GTSRB, SVHN, and Aircraft? This would help distinguish the benefit of non-uniform spatial weighting from that of image-dependent attention.

  3. Are the Table 1 classification results based on single training seeds? Please report variability across seeds for a representative subset, particularly where margins are below 1 pp. A controlled VQT comparison using the same feature-noise and batch-size settings would also help assess whether training differences contribute to these margins.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.