- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
HIFA combines intermediate representations from frozen vision foundation models through intra-backbone and inter-backbone attention. It achieves 71.9% on VTAB-1k, compared with 66.0% for final-layer concatenation. Ablations suggest that learned fusion explains much of the gain, while additional benefits from intermediate depth are task-dependent.
Critical strengths
- The two-stage fusion architecture is clearly structured, and the experiments demonstrate HIFA’s effectiveness, supported by comprehensive ablations and well-designed controls.
Critical weaknesses
- The benefits of additional complexity are insufficiently justified. Three-encoder HIFA improves over DINOv3 intra-only by approximately one percentage point while requiring two additional encoders and roughly three times as many fusion-head parameters.
- The methodological novelty appears limited. The method primarily combines existing intermediate-layer aggregation and cross-model attention into a two-stage architecture.
Questions for authors
- Including feature extraction costs, how does the three-encoder system compare with single- and two-encoder alternatives?
End-to-End Inference Costs and the Value of Additional Encoders
End-to-end computational cost. To address the concern about the cost of additional frozen encoders, we conducted new end-to-end inference measurements comparing single-, two-, and three-encoder configurations. These measurements include the entire inference pipeline rather than only fusion-head computation.
We benchmarked DINOv3 intra-only, DINOv3+MAE, and full three-encoder HIFA on an NVIDIA H100 NVL (FP32, 224×224 images, 1,000 distinct SUN397 examples, three timed repetitions, batch sizes 1 and 32). These are online runs without cached features: timing includes preprocessing, frozen encoder forwards and intermediate pooling, transfers, and fusion. Checkpoint loading is excluded.
At batch size 1:
The accuracy figures are the prior three-seed means across all 19 VTAB-1k tasks, not accuracies measured on the SUN397 timing subset. Fusion parameter counts exclude the task classifier. Table 7 and Appendix N report the complete measurements and variability. At batch size 32, end-to-end latency is 22.5, 31.0, and 45.2 ms/image, respectively.
What does adding each encoder contribute? Adding MAE to DINOv3 increases mean accuracy from 0.709 to 0.722, at a cost of 20.2 additional ms/image and 1,169 MiB peak allocated GPU memory (batch 1).
Adding I-JEPA to that pair increases latency by another 30.8 ms/image and peak allocated memory by 4,015 MiB, while mean accuracy moves from 0.722 to 0.719. The paired trio-minus-pair difference is not statistically significant (mean -0.0030; 95% bootstrap CI [-0.0071, +0.0006]; Wilcoxon p=0.352).
Relative to the trio, DINOv3+MAE reduces batch-1 latency by approximately 28% and peak GPU memory by 47%, while retaining the numerically highest mean accuracy among the evaluated configurations.
The fusion head accounts for less than 1% of end-to-end latency. At batch size 1, frozen encoder forwards account for roughly 49-63%, so reducing fusion-head parameters alone would not address the main cost of adding encoders. These observations are specific to the measured hardware and data conditions, not a universal efficiency ranking.
Practical implications. The three-encoder configuration should not be considered the default simply because the architecture supports it. DINOv3+MAE offers the most favorable observed accuracy-efficiency trade-off among the evaluated configurations. Other encoder combinations and tasks may behave differently, so encoder selection should account for both validation performance and computational constraints.
Methodological contribution. HIFA does not introduce attention aggregation or cross-attention as new primitives. Instead, it explicitly separates within-encoder depth aggregation from cross-encoder fusion using compact layer descriptors.
The experiments also investigate what actually drives the improvement. Conventional final-feature concatenation reaches 0.660; packed-final concatenation reaches 0.671; learned final-token processing reaches 0.702-0.704; and full HIFA reaches 0.719. The additional numerical effect of intermediate-depth access is task-dependent and not statistically established across all 19 tasks.
Single-encoder and retrained-pair evaluations further demonstrate that adding backbones does not necessarily improve accuracy. Attention allocation and test-time masking complement these results by characterizing how the trained model uses different depth regions and encoder summaries, without treating attention weights as causal proof.
The revised title, Understanding Hierarchical Fusion Across Vision Foundation Models, reflects this focus on understanding the contribution and cost of each information source rather than attributing all improvements to hierarchical complexity.