acceptodds
Under review as a conference paper at ICLR 2027

LEARNING HIERARCHICAL FUSION OF INTERMEDIATE REPRESENTATIONS ACROSS VISION FOUNDATION MODELS

Abstract

Modern vision foundation models expose rich hierarchies of intermediate rep- resentations, yet downstream transfer commonly reduces each encoder to its final feature; this discards potentially useful information across depth within an encoder and across independently trained encoders. We introduce Hierarchical Fusion At- tention (HIFA), a hierarchical attention mechanism that first aggregates interme- diate representations within each frozen vision backbone and then fuses the result- ing summaries across backbones, operating on compact global layer descriptors rather than dense spatial maps so as to separate the selection of useful representa- tion levels from the combination of heterogeneous encoders. On VTAB-1k, HIFA achieves substantially higher mean accuracy than concatenation of final-layer rep- resentations, with particularly large gains on Structured tasks: conventional con- cat reaches 0.660, a packed-final concat control 0.671, final-only HIFA 0.702, an architecture-matched final-token-only HIFA 0.704, and full HIFA 0.719. The near-equality of final-only and matched final-token indicates that the remaining numerical lift to full HIFA is associated with access to intermediate depth under the same hierarchical architecture; this depth-related difference is numerical and task-dependent rather than a statistically established uniform cross-task improve- ment. Beyond predictive performance, intra-backbone attention reveals alloca- tion across network depth and inter-backbone attention reveals encoder contribu- tions; combined with test-time masking interventions, these analyses show that the trained fusion mechanism depends strongly on specific depths and encoder summaries.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
vision foundation models, intermediate representations, representation fusion, multi-backbone fusion, transfer learning, attention, representation learning, frozen encoders, VTAB-1k

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AqyiOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HIFA combines intermediate representations from frozen vision foundation models through intra-backbone and inter-backbone attention. It achieves 71.9% on VTAB-1k, compared with 66.0% for final-layer concatenation. Ablations suggest that learned fusion explains much of the gain, while additional benefits from intermediate depth are task-dependent.

Critical strengths
  1. The two-stage fusion architecture is clearly structured, and the experiments demonstrate HIFA’s effectiveness, supported by comprehensive ablations and well-designed controls.
Critical weaknesses
  1. The benefits of additional complexity are insufficiently justified. Three-encoder HIFA improves over DINOv3 intra-only by approximately one percentage point while requiring two additional encoders and roughly three times as many fusion-head parameters.
  2. The methodological novelty appears limited. The method primarily combines existing intermediate-layer aggregation and cross-model attention into a two-stage architecture.
Questions for authors
  1. Including feature extraction costs, how does the three-encoder system compare with single- and two-encoder alternatives?
1 comment
AuthorsOpenReview ↗

End-to-End Inference Costs and the Value of Additional Encoders

End-to-end computational cost. To address the concern about the cost of additional frozen encoders, we conducted new end-to-end inference measurements comparing single-, two-, and three-encoder configurations. These measurements include the entire inference pipeline rather than only fusion-head computation.

We benchmarked DINOv3 intra-only, DINOv3+MAE, and full three-encoder HIFA on an NVIDIA H100 NVL (FP32, 224×224 images, 1,000 distinct SUN397 examples, three timed repetitions, batch sizes 1 and 32). These are online runs without cached features: timing includes preprocessing, frozen encoder forwards and intermediate pooling, transfers, and fusion. Checkpoint loading is excluded.

At batch size 1:

ConfigurationVTAB-1kms/imagePeak GPU MiBFusion params
DINOv3 intra-only0.70957.83,3922.38M
DINOv3+MAE0.72278.04,5615.55M
Full HIFA0.719108.88,5767.34M

The accuracy figures are the prior three-seed means across all 19 VTAB-1k tasks, not accuracies measured on the SUN397 timing subset. Fusion parameter counts exclude the task classifier. Table 7 and Appendix N report the complete measurements and variability. At batch size 32, end-to-end latency is 22.5, 31.0, and 45.2 ms/image, respectively.

What does adding each encoder contribute? Adding MAE to DINOv3 increases mean accuracy from 0.709 to 0.722, at a cost of 20.2 additional ms/image and 1,169 MiB peak allocated GPU memory (batch 1).

Adding I-JEPA to that pair increases latency by another 30.8 ms/image and peak allocated memory by 4,015 MiB, while mean accuracy moves from 0.722 to 0.719. The paired trio-minus-pair difference is not statistically significant (mean -0.0030; 95% bootstrap CI [-0.0071, +0.0006]; Wilcoxon p=0.352).

Relative to the trio, DINOv3+MAE reduces batch-1 latency by approximately 28% and peak GPU memory by 47%, while retaining the numerically highest mean accuracy among the evaluated configurations.

The fusion head accounts for less than 1% of end-to-end latency. At batch size 1, frozen encoder forwards account for roughly 49-63%, so reducing fusion-head parameters alone would not address the main cost of adding encoders. These observations are specific to the measured hardware and data conditions, not a universal efficiency ranking.

Practical implications. The three-encoder configuration should not be considered the default simply because the architecture supports it. DINOv3+MAE offers the most favorable observed accuracy-efficiency trade-off among the evaluated configurations. Other encoder combinations and tasks may behave differently, so encoder selection should account for both validation performance and computational constraints.

Methodological contribution. HIFA does not introduce attention aggregation or cross-attention as new primitives. Instead, it explicitly separates within-encoder depth aggregation from cross-encoder fusion using compact layer descriptors.

The experiments also investigate what actually drives the improvement. Conventional final-feature concatenation reaches 0.660; packed-final concatenation reaches 0.671; learned final-token processing reaches 0.702-0.704; and full HIFA reaches 0.719. The additional numerical effect of intermediate-depth access is task-dependent and not statistically established across all 19 tasks.

Single-encoder and retrained-pair evaluations further demonstrate that adding backbones does not necessarily improve accuracy. Attention allocation and test-time masking complement these results by characterizing how the trained model uses different depth regions and encoder summaries, without treating attention weights as causal proof.

The revised title, Understanding Hierarchical Fusion Across Vision Foundation Models, reflects this focus on understanding the contribution and cost of each information source rather than attributing all improvements to hierarchical complexity.

Reviewer_RDAAOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, the authors propose HIFA, a hierarchical attention mechanism consisting of two steps: 1) fusion of summary representations across intermediate representations within a backbone, and 2) fusion of per-backbone representations. The authors evaluate their proposed method on VTAB-1k and show that the hierarchical inter- and intra-backbone fusion improves over linear probing of individual backbones as well as confidence-based voting and concatenation.

Critical strengths

None

Critical weaknesses
  1. The paper has limited methodological novelty. The main contribution is a hierarchical architecture for fusing intermediate representations across multiple vision foundation models. However, attentive fusion of intermediate representations has already been explored in prior work (e.g., ALF, InCA), and ComBo has demonstrated the fusion of intermediate representations across multiple backbones. The proposed hierarchical factorization therefore appears to be a relatively incremental architectural contribution.

  2. The evaluation lacks comparisons with established methods that address closely related problems. In particular, the authors should demonstrate that HIFA (1) improves over advanced single-backbone probing methods that fuse intermediate representations (e.g., ALF) and (2) offers advantages over existing multi-backbone fusion approaches (e.g., ComBo). Instead, the evaluation mainly considers relatively simple baselines, such as linear probing and final-layer concatenation, alongside ablations of HIFA itself. These comparisons are insufficient to establish the benefits of HIFA over existing methods, especially given the overlap with prior work.

  3. The paper presents insufficient evidence for the benefits of hierarchical fusion. The hierarchical factorization is the central architectural contribution, yet its advantages over simpler fusion mechanisms are not convincingly demonstrated. The improvements over flat attention and weighted-sum aggregation are not statistically significant (App. F). Furthermore, these alternatives use hyperparameters selected for HIFA rather than being tuned independently, and the comparisons are not matched for parameter count or search budget. It therefore remains unclear whether the proposed hierarchy offers meaningful benefits over simpler alternatives.

  4. The main results report only mean accuracies across three random seeds and multiple benchmarks within each VTAB category, without standard deviations or confidence intervals. This makes it difficult to assess the reliability of the reported improvements, especially given the small differences between several methods. While the authors provide statistical analyses across tasks in the appendices, variability across seeds is not clearly reported.

  5. The experimental methodology and results are difficult to follow. While the motivation and proposed architecture are reasonably clear, the experimental sections introduce numerous closely related variants whose differences are difficult to distinguish. Important details, such as what the authors refer to as “architecture search,” are also scattered across the paper and appendices. The presentation of results is similarly fragmented. This is particularly apparent in Section 7, where the discussion largely consists of abbreviated statements and numerical results rather than a coherent interpretation of the findings. Furthermore, the appendix contains references to .csv artifacts rather than presenting the corresponding results in a self-contained manner. Overall, parts of the manuscript read more like preliminary notes than a finished scientific paper, giving the impression that it has not undergone a thorough final revision.

Questions for authors
  1. Given the close relationship to existing methods such as ALF and ComBo, could the authors clarify what advantages HIFA offers over these approaches? In particular, can they provide empirical comparisons using matched backbone configurations and evaluation protocols to determine whether the proposed method improves upon existing intermediate-representation fusion methods?
  2. Given that the improvements over flat attention and weighted-sum aggregation are not statistically significant (Appendix F), what evidence supports the necessity of the proposed hierarchical architecture? Could the authors provide comparisons in which the alternative fusion mechanisms receive independent hyperparameter tuning and are matched for parameter count and search budget?
1 comment
AuthorsOpenReview ↗

Clarifying the Contribution and Evidence for Hierarchical Fusion

Contribution and novelty. The review raises two important questions: what distinguishes HIFA from existing fusion methods, and what evidence supports its hierarchical design. We address these separately because they concern different aspects of the work.

We do not claim that intermediate-layer attention or cross-model attention is new. ALF and InCA mainly investigate intermediate representations within an encoder, while ComBo is a relevant multi-encoder precedent that retains dense spatial features. HIFA instead represents each native layer by one compact global descriptor, first aggregates depth within each backbone, and then fuses the resulting backbone summaries.

The contribution is this explicit factorization together with an empirical investigation of where fusion gains originate. We revised the title, abstract, related work, and discussion to make this positioning clear.

What drives the reported improvement? We now present a stepwise comparison: conventional final-feature concat (0.660), packed-final concat using HIFA's final descriptors (0.671), final-only HIFA (0.702), architecture-matched final-token HIFA (0.704), and full HIFA (0.719).

Enriching final descriptors under a linear classifier accounts for only 0.011, while learned processing of final features recovers considerably more. Opening intermediate depth within the selected HIFA architecture adds a further numerical 0.015, but that difference is heterogeneous across tasks (paired Wilcoxon p=0.396). We therefore do not attribute the entire improvement to intermediate representations.

Evidence for hierarchical factorization. The existing results do not establish that hierarchy is necessary. HIFA reaches 0.719, compared with flat attention at 0.705, weighted-sum at 0.704, and inter-linear at 0.714. Their paired Wilcoxon p-values are 0.123, 0.123, and 0.709, respectively.

Flat and weighted-sum also have smaller fusion heads (4.19M and 4.13M parameters versus 7.34M), inherited HIFA's selected hyperparameters, and received no independent tuning. These limitations are now explicitly documented in Sections 4.4 and 7 and Appendix H.

Nevertheless, fixed-configuration ablations answer a valid, narrower question: how does performance change when a selected component is replaced while the remaining configuration and training protocol are held fixed? Independently tuned, parameter- and search-budget-matched alternatives would address the different question of relative architectural competitiveness.

We do not present our ablations as a substitute for that stronger comparison. Likewise, the matched final-token control instantiates the same 7.34M HIFA modules but restricts layer access. It provides a targeted comparison of intermediate-depth access within HIFA, not capacity fairness between different fusion families.

Relationship to ALF and ComBo. Evaluations with matched backbones and protocols would strengthen claims about relative performance. Such experiments have not been conducted, and the revised manuscript makes no superiority claim over either method.

Table 6 reports only unmatched analytical fusion-input payload (0.774 versus 28.90 MB/image) and fusion-head compute (0.155 versus 1.217 GMAC), excluding frozen encoder forwards. These estimates are not end-to-end latency or peak-memory comparisons. Appendix O further shows that transferring HIFA's selected architectural settings to another encoder suite yields 0.689 overall, demonstrating sensitivity to encoder composition rather than guaranteed transferability.

The controlled within-HIFA analyses address a different scientific question from whether HIFA outperforms established methods under matched conditions. We explicitly separate these conclusions.

Statistical evidence and presentation. Tables 3-5 now report mean±SD over three seeds. Appendices F-G provide paired bootstrap intervals, Wilcoxon tests, and task-level effects. Table 2 distinguishes the structural variants; Section 4.3 and Appendix B specify the sequential, non-exhaustive architecture search; and Appendix H audits which controls inherited its settings.

We also rewrote Section 7 around the interpretation and limitations of the findings, and incorporated key numerical analyses directly into manuscript tables rather than relying solely on external CSV references.

The revised contribution is therefore a compact hierarchical fusion interface and a transparent empirical decomposition showing that learned processing of final features accounts for substantial gains, while intermediate depth and additional encoders offer conditional rather than universal benefits. These findings remain informative without establishing superiority over independently optimized fusion architectures.

Reviewer_YGRQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces HIFA, a method to combine features from multiple frozen vision models. It uses two levels of attention: first, it summarizes the intermediate layers of each model into a compact representation, and then it fuses these summaries across different models. The main contribution is a detailed analysis of where the performance gains come from on the VTAB-1k benchmark. The authors show a clear progression of scores from simply concatenating final features to their full hierarchical model. They also honestly report that many of the improvements are not statistically significant.

Critical strengths
  • The paper addresses a practical and important problem. When using multiple frozen vision models, simply concatenating all intermediate features becomes very expensive. The authors clearly motivate the need to understand exactly where the performance gains come from when fusing these models.

  • The two-level design is clean and well-structured. It first aggregates layers within one model and then fuses information across different models. Sharing the architecture design across different models without tying weights makes it very easy to add new models later.

  • The step-by-step analysis of the results is very transparent. The authors carefully separate the gains from better final features, learned fusion, and intermediate depth.

  • The authors are very honest about their results. They clearly show that the gain from intermediate depth is small and not statistically significant across the full benchmark. Reporting negative or weak results this clearly is highly commendable.

Critical weaknesses
  • The main finding of the paper is that final-layer fusion captures most of the performance, and intermediate depth adds very little. However, the title and abstract still heavily emphasize the "hierarchical" and "intermediate" aspects, which makes the framing a bit misleading.

  • The baseline methods, such as simple weighted sums or flat attention, are not matched in parameter count or search budget to the proposed method. Because of this, it is hard to tell if the gains come from the hierarchical design itself or just from having more trainable parameters.

  • Some design choices are not fully explained. For example, the authors mention using a spatial basis for the queries but do not deeply analyze if this basis should be learned or fixed, or how it behaves across different layers.

  • The evaluation is only done on the VTAB-1k benchmark. To prove the method works broadly, it should also be tested on dense prediction tasks like segmentation or retrieval tasks.

  • The efficiency comparison with the ComBo method is not completely fair. The two methods use different sets of encoders and different adapter types, so the comparison mixes architecture differences with encoder differences.

Questions for authors
  • Can the authors provide the standard deviations across multiple random seeds for the main results to show how stable the numbers are?

  • If the linear classifier in the packed-final baseline is matched in parameter count to the proposed method, does the small performance gain still exist?

  • Since the DINO+MAE pair performs as well as the three-model combination, what is the practical advice for a user to choose which two models to use without testing all pairs?

  • How sensitive is the selected architecture to the choice of the six development tasks? Does it change if the development tasks are resampled?

1 comment
AuthorsOpenReview ↗

Clarifications on Framing, Baseline Fairness, and Additional Evidence

Framing and contribution. We agree that the original manuscript placed too much emphasis on intermediate-depth gains. We changed the title to Understanding Hierarchical Fusion Across Vision Foundation Models and revised the abstract and discussion to reflect what our experiments establish.

Our main question is where the gains of multi-backbone fusion come from, rather than whether hierarchical attention is universally necessary. Conventional final-feature concatenation reaches 0.660, packed-final concat 0.671, final-only HIFA 0.702, architecture-matched final-token HIFA 0.704, and full HIFA 0.719. Learned processing of final descriptors accounts for much of the improvement. Access to intermediate depth adds a task-dependent numerical 1.5 percentage points within HIFA, but the paired 19-task test does not establish a statistically significant advantage (Wilcoxon p=0.396).

Fairness of the controls. Flat attention and weighted-sum inherited HIFA's selected hyperparameters and were not independently tuned or matched for parameter count or search budget. We now disclose this explicitly (Secs. 4.4 and 7; App. H). These ablations assess the effect of replacing a component under a fixed experimental configuration; they do not establish which fusion family would perform best after independent optimization.

The matched final-token control instantiates the same 7.34M-parameter HIFA modules but restricts access to intermediate memory. It addresses depth within the selected architecture, not capacity-matched superiority over other fusion families.

Regarding the packed-final question, a strictly linear classifier from fixed 6,016-dimensional descriptors to C classes has 6,016C+C parameters. Matching HIFA's parameter count would require changing its parameterization. An MLP, for example, would introduce nonlinear capacity and would no longer constitute a linear probe. We have not evaluated that alternative, and acknowledge that such a comparison would provide additional evidence about capacity.

Query design. Section 3.2 clarifies that HIFA does not use a fixed spatial query basis. Each backbone has an independently learned query embedding (Q=1 in the selected configuration), shared across images, while attention weights depend on image-specific layer keys and values. Appendix B documents the query-design alternatives considered.

Variability and search sensitivity. Tables 3-5 now report mean and sample SD over seeds 41/42/43 (HIFA: 0.719 ± 0.007). Appendices F-G provide paired tests, confidence intervals, and task-level depth effects.

Retrospective reweighting of historical development trials shows that all seven axis-level winners remain unchanged when any one of the six development tasks is removed. Under leave-one-VTAB-group-out reweighting, six of seven remain stable; removing the Structured group changes the preferred intra-attention configuration from 1×4 to 1×8. This measures historical candidate-ranking stability, not whether rerunning the sequential search would recover the same architecture.

Choosing encoders and measuring cost. We retrained all three backbone pairs. DINOv3+MAE is the strongest evaluated pair (0.722 versus 0.719 for the trio; paired Wilcoxon p=0.352). New uncached, end-to-end inference measurements on an H100 NVL give 78.0 versus 108.8 ms/image and 4,561 versus 8,576 MiB peak allocated GPU memory at batch size 1 (Table 7; App. N).

Our practical recommendation is to start from strong individual encoders and validate promising subsets against accuracy and resource constraints, rather than assume that adding encoders will help. We do not claim a universal selection rule that avoids evaluating candidate pairs.

ComBo and broader tasks. Different encoder suites and evaluation protocols prevent a fair accuracy or end-to-end efficiency ranking against ComBo. Table 6 is now explicitly identified as an unmatched analytical fusion-footprint comparison, not evidence of superior overall efficiency.

Our demonstrated scope is VTAB-1k classification using compact global descriptors. Segmentation and retrieval remain untested and may require different representation granularity.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.