acceptodds
Under review as a conference paper at ICLR 2027

Do Hierarchical Vision Backbones Need a Fourth Stage?

Abstract

Hierarchical vision backbones devote substantial capacity to their final stage. We investigate how much of this capacity is needed and how learning without it changes the preceding hierarchy. We train Swin and ConvNeXt models from Nano to Base with and without the fourth-stage blocks, always keeping the final downsampling transition. Small and Base models trained without these blocks use about 29% fewer parameters and 25% less peak inference memory at batch size one, and lose only 0.13–0.53 ImageNet-1K Top-1 points. Reallocating this capacity upstream can improve accuracy at comparable parameter counts, at the cost of more FLOPs, whereas the narrowest configurations still favor four stages. Under a common normalized readout, class information becomes accessible earlier within the third stage in all ten model pairs, and truncated models generally depend more on the late spatial exchanges of this stage. On Swin-T-C64, a terminal token-mixing module with 8.6% of the original stage's parameters recovers about half of the truncation loss and outperforms a parameter-matched channel-only control. COCO box and mask AP stay within one point of the full models in the evaluated configurations, and ImageNet-C gaps are modest at Small and Base scale but larger in some smaller configurations. These findings support choosing terminal capacity jointly with upstream allocation and the evaluation objective.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
hierarchical vision backbones, fourth stage, capacity allocation, representation adaptation, robustness

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6PVfOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper tries to answer an interesting question: Do Hierarchical Vision Backbones Need a Fourth Stage? It conducts extensive experiments across different downstream tasks, model families, and model scales.

Critical strengths
  1. The research question sounds interesting.
  2. The paper provides comprehensive and detailed experiments and makes a substantial effort to discuss the findings thoroughly.
Critical weaknesses

The only but fundamental concern is that all the experiments in this paper can be explained by a familiar relationship between model capacity and data demands, rather than establish the architectural value or necessity of the fourth stage itself. Capacity redundancy is already widely understood. Observing it does not constitute a sufficiently novel or valuable work. The experiments do not sufficiently rule out this simpler explanation. Therefore, it remains unclear whether the results reflect a distinct architectural role of the fourth stage itself.

For example, the paper’s central observation is that removing the fourth stage has a smaller impact on ImageNet-1K performance as Swin and ConvNeXt become larger. However, this observation is fully consistent with a capacity-based explanation: removing S4 from narrow models eliminates a substantial portion of their capacity, leading to a drop of 2–3 percentage points, whereas Small/Base models lose only 0.13–0.53 points because the preceding stages may retain sufficient capacity for the task. These results therefore do not distinguish the contribution of the fourth-stage architecture from the effect of overall model capacity.

After reviewing all the experimental designs and results, the reviewer could not identify enough evidence that goes beyond the "capacity redundancy" explanation. The concern is whether this paper provides new insight into the fourth stage itself. However, the reviewer is open to discuss about it.

In addition, the reviewer finds the paper's organization and presentation confusing. The Appendix contains 18 sections and a large number of experimental results, while the main paper occupies only eight and a half pages and does not appear to be carefully formatted.

Questions for authors

Please address the concern raised in the weaknesses and provide additional experiments to support the response. Please discuss the function of fourth stage and analyze why the fourth stage would be not necessary in some cases.

Reviewer_ETgjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents a study examining how various performance metrics (number of parameters, accuracy, peak memory, and others listed in the appendix, such as FLOPs) change when the fourth stage of a hierarchical vision backbone is removed and the architecture modified; see the three proposed variants (ii through iv) for this purpose. This study is conducted on the Swin and ConvNeXt models (which are standard models for studies on hierarchical training) for architectures ranging from the Nano to the Base scale. The (long) appendix describes the experiments in greater detail and also presents additional results.

Critical strengths
  • [S1] The main strength lies in the systematic study, conducted through three proposed architectural modifications (ii through iv, as shown in Figure 1, with the last two appearing to be original), to test the usefulness of the fourth stage. There is an impressive number of experiments, each of which provides insights.

  • [s2] The authors took the time to test their approach on three benchmarks (ImageNet-C, ImageNetV2, and ImageNet-ReaL); see Table 4. We can see that the truncated approach for the Base versions achieves an accuracy close to that of the full model. Similar results on COCO are provided in Table 5.

Critical weaknesses

[W1] The paper’s greatest weakness is probably that some results are already known to the structured pruning community and that it is primarily a paper that reports on a series of practical experiments (note that the “Reproducibility Statement” section does not mention the code availability or the variability of the results). Also, the problem with a purely experimental approach is that it becomes sensitive to side effects. For example, training is performed on ImageNet-1K, for classification at 224 × 224 resolution (Russakovsky et al., 2015). While this dataset is the de facto standard, Northcutt, Athalye, and Mueller [NAM21] found that it accounts for 6% of the ImageNet validation set. Similarly, Yun et al. [Yun+21] highlights the problems with ImageNet-1K, particularly regarding the impact of errors in the training set. Indirectly, this issue is partially addressed through the use of partially cleaned benchmarks in some results (such as ImageNet-Real, but not ImageNet-V2).

[w2] One might also question the relevance of the representation measurement as defined by Equation (3). The three major objections are the effect of discretization, the appropriateness of evaluating a representation via accuracy, and, in the case of this paper, the effect of label noise, which potentially affects the stages of a backbone differently.

Misc.

  • I suggest increasing the font size in Figure 1.
  • The discussion regarding structured pruning could be expanded (Section 2.2), particularly by addressing data-independent vs. data-dependent methods. This, incidentally, supports one of the authors’ conclusions when they discuss objective evaluation.
  • The behavior of Tiny architectures in terms of representation is intriguing (see Figure 2, g-h).

[NAM21] Curtis Northcutt, Anish Athalye, and Jonas Mueller. “Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks.” In: Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks. Vol. 1. Virtual conference: Curran Associates, Inc., Dec. 2021, pp. 1–13.

[Yun+21] Sangdoo Yun, Seong Joon Oh, Byeongho Heo, Dongyoon Han, Junsuk Choe, and Sanghyuk Chun. “Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels.” In: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Nashville, Tennessee, USA: IEEE, June 2021, pp. 2340–2350.

Questions for authors

[Q1] My main, scientific, question is about the impact of label noise on the analysis. In fact, there are several papers that explain that if a model is pruned, it becomes less sensitive to label noise and might therefore outperform a full model on a cleaned dataset (see, for example, Jin et al. [Jin+22] and He et al. [He+22]).

[Q2] Another question concerns the potential impact of the paper’s findings. It’s not entirely clear to me how the central discussion on S4 impacts the community today. In other words, please clarify how far the findings go beyond current knowledge.

[Jin+22] Tian Jin, Michael Carbin, Daniel M. Roy, Jonathan Frankle, and Gintare Karolina Dziugaite. “Pruning’s Effect on Generalization Through the Lens of Training and Regularization.” In: Advances in Neural Information Processing Systems (NeurIPS). Vol. 35. New Orleans, Louisiana, USA: Curran Associates, Inc., Nov. 2022, pp. 1–15.

[He+22] Zheng He, Zeke Xie, Quanzhi Zhu, and Zengchang Qin. “Sparse Double Descent: Where Network Pruning Aggravates Overfitting.” In: International Conference on Machine Learning (ICML). Vol. 162. Proceedings of Machine Learning Research. ML Research Press, July 2022, pp. 8635–8659.

1 comment

Thank you for recognizing the systematic investigation and the insights provided by the experiments. We have clarified the paper’s objective: explaining when S4 contributes useful computation, which upstream changes can substitute for it, and how the remaining hierarchy adapts.

  • Contribution beyond pruning results (W1, Q2). Preserving accuracy with fewer parameters is established, and HiViT demonstrates competitive three-stage architectures. Our contribution concerns the conditions behind that possibility. We cross local/global S3 mixing with S4 presence at fixed S3 depth and width, retaining the final downsampling. These controls show that upstream spatial computation changes the benefit of S4. Shortening global S3 to 14 blocks then allows Swin-S to reach 83.10% versus 83.04% for its full local reference, with approximately 43% fewer parameters and 21% fewer FLOPs in the reported runs. A deeper local alternative remains less accurate despite using more resources. The tested ConvNeXt kernel enlargements do not reproduce this recovery. These results identify architectural conditions and limits beyond a catalogue of compression trade-offs.

  • Label noise and ReaL (W1, Q1). Training-label noise and evaluation-label errors must be distinguished. Northcutt et al. examine evaluation annotations, whereas Yun et al. also address training supervision. ReaL revises evaluation labels without correcting our training labels. Nevertheless, it directly informs the proposed ranking reversal: the full model remains more accurate in all ten local full/truncated pairs under ReaL, although the truncation penalty decreases. Thus, truncated models do not overtake their full counterparts under reassessed labels. The cited pruning studies also support a conditional interpretation: Jin et al. distinguish training and regularization effects, while He et al. show that pruning can mitigate or aggravate overfitting depending on sparsity. These findings do not establish a general prediction for S4 removal. Our results constrain the proposed explanation without isolating sensitivity to training-label noise.

  • Interpretation of the representation measure (W2). The probe measures class accessibility under a specified readout, not representation quality generally. Top-1 ignores confidence and margins, and probe fitting and evaluation remain label-dependent. For depth discretization, every S3 block is evaluated, with identical block positions within each local full/truncated pair. Equation (3) summarizes the profiles through piecewise-linear interpolation after subtracting the input difference; it is a descriptive statistic, not a discretization-invariant quantity. Crucially, the Nano allocation controls show stronger intermediate probe accuracy alongside worse native accuracy at comparable parameter budgets. Fixed-weight spatial restrictions separately examine functional dependence. Together, these analyses distinguish accessible information from the computations supporting predictions.

  • Relation to structured pruning. The revised related work distinguishes criteria using learned weights alone from those requiring activations, gradients, or evaluation data. Weight-only criteria still inherit dependence on training data. Our removal target is prescribed by the architectural question: independent training tests what the modified architecture can learn, whereas fixed-weight interventions examine reliance in an existing model.

  • Variability and reproducibility. The revised manuscript reports three training runs per local full/truncated architecture except standard Tiny. Global and shortened variants are identified as individual runs. Probe intervals describe image-level uncertainty conditional on the checkpoints, separately from training variability. We will explicitly state code availability in the reproducibility statement.

  • Tiny profiles and presentation. Standard Tiny and Tiny-C64 differ in depth and allocation, so their profiles cannot be interpreted as a width-only effect. Proximity to the classifier may also contribute to earlier decodability. The revised interpretation emphasizes these distinctions alongside functional dependence. The original overview figure has been replaced, with explicit labels distinguishing full/truncated models and allocation controls.

We hope these revisions clarify the explanatory contribution and the scope of the architectural and representation evidence.

Reviewer_WdyqOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Most hierarchical vision backbones use four stages, with the final stage (S4) holding a significant number of the parameters. The authors train Swin and ConvNeXt models from Nano to Base scale with and without the S4 blocks, and study the accuracy and efficiency trade-off, the effect of reallocating S4 capacity to S3, how the S3 representations change when S4 is absent during training, and whether a lightweight terminal module can recover the lost accuracy.

Critical strengths

The main contribution is that of a more controlled investigation into the importance of the S4 Blocks in hierarchical vision backbones than what previous works have done. Their main findings indicate that the S4 blocks can sometimes be dropped without a significant reduction in accuracy when training Swin/ConvNeXt Nano to Base (∆ Top-1 of -3.08 % to -0.13%) on ImageNet.

Critical weaknesses

Overall, I found the paper to have shortcomings (listed below):

  1. ImageNet-1K is a very saturated benchmark. In Table S3, Swin-S is only 0.29 points below Swin-B while having about 43% fewer parameters. The small accuracy drop after removing S4 could therefore just reflect these diminishing returns rather than S4 being redundant. I would recommend additional transfer learning experiments (with and without S4 blocks after ImageNet pretraining) on fine-grained classification datasets to see if there is an accuracy gap between full and truncated models there.
  2. The CoCo transfer learning results seem to cause a fairly significant drop in performance, especially when we consider that the parameter savings will be lower for a Mask R-CNN architecture. I also suggest adding parameter counts / FLOPs for this.
  3. The S4 truncation only modestly reduces the FLOPs (about 10% on average) and improves throughput by about the same (table S3.b). The largest benefit is the savings in peak memory, which really is the upper limit reported as the batch size is set to one.
  4. HiViT [1] reports 83.6% Top-1 for a three-stage Swin-B with 66.3M parameters. This exceeds the authors own four-stage Swin-B (83.33%, 87.7M parameters). If a better three-stage design already exists, what does a practitioner gain from this study?
  5. In table 4, the full Swin-S is better than or equal to the truncated Swin-B on ImageNet-C (58.00 vs 57.37), ImageNet-v2 (72.37 vs 71.92), and ReaL (87.76 vs 87.76), while using fewer parameters (49.6M vs 62.5M), fewer FLOPs (8.74 vs 14.19 G), less peak memory, and achieving higher throughput (table S3). The truncated model leads only on ImageNet, by 0.15 points. This suggests that at least sometimes choosing a smaller full model is a better trade-off than truncating a larger one, which undermines the practical motivation for removing S4 and this is never discussed.
  6. The finding that class information becomes decodable earlier within S3 (Figure 2) is expected: removing S4 places every S3 block closer to the classification head, and representations closer to the output are generally more class-specific [2]. The authors should reconsider or better motivate why these experiments are worthwhile including.
  7. The lightweight terminal module is listed as a main contribution but it is evaluated only on Swin-T-C64, these experiments should be expanded if they’re to be a main contribution.

References

[1] Xiaosong Zhang, Yunjie Tian, Lingxi Xie, Wei Huang, Qi Dai, Qixiang Ye, and Qi Tian. HiViT: A simpler and more efficient design of hierarchical vision transformer. In International Conference on Learning Representations, 2023.
[2] Alain, Guillaume, and Yoshua Bengio. "Understanding intermediate layers using linear classifier probes." arXiv preprint arXiv:1610.01644 (2016).

Questions for authors

The CoCo transfer learning results seem to cause a fairly significant drop in performance, especially when we consider that the parameter savings will be lower for a Mask R-CNN architecture. Could you please provide an explanation for the truncated model's behavior on such tasks?

1 comment

Thank you for your careful review and for recognizing the value of a controlled investigation of S4. We have revised the manuscript to clarify its central objective: explaining what makes the final stage useful, when upstream computation can substitute for it, and how the remaining hierarchy adapts.

  • Contribution relative to HiViT. HiViT provides a successful three-stage design and valuable architectural insights. Its results motivate our investigation: the existence of a competitive three-stage model leaves open which changes enable stage removal and where they remain effective. We independently vary S3 mixing and S4 presence while retaining S3 depth, width, and the final downsampling, separating changes combined in HiViT's architectural transition. We then examine further depth reduction and the adaptation of the learned representations. The contribution is this additional explanatory evidence, rather than a claim of a superior backbone. For practitioners, it identifies conditions under which removing S4 works and architectural settings where the tested adaptations fail.

  • ImageNet saturation and diminishing returns. The revised evidence goes beyond comparing model sizes. At fixed S3 depth and width, global mixing improves truncated Swin-S from 82.50% to 83.45%, with only 0.34% more parameters. Shortening the global S3 to 14 blocks then achieves 83.10%, compared with 83.04% for the full local reference, with approximately 43% fewer parameters and 21% fewer FLOPs in the reported runs. A deeper, 20-block local alternative reaches 82.54% despite using more parameters and computation. These controls show that the organization of spatial computation matters beyond the width-related diminishing returns. The tested ConvNeXt kernel enlargements do not reproduce this recovery, further delimiting the result. Fine-grained transfer remains outside our current evidence.

  • COCO transfer and detector costs. The observed losses are 0.3–0.9 box AP and 0.5–0.9 mask AP after 24 epochs across the four pairs. Retaining the final downsampling preserves the FPN interface, but removes the subsequent S4 processing of its coarsest feature map. Matching feature dimensions therefore does not ensure equivalent detection features. These results establish a transfer limitation without identifying a unique causal mechanism. Appendix A.8 reports analytical whole-detector estimates, including the FPN and heads: 15.1–20.5% fewer parameters and 2.6–4.2% fewer FLOPs. These evaluations concern local truncation, not the reconstructed global variants.

  • Practical savings and smaller full models. Your comparison between full Swin-S and truncated Swin-B is valid and is now explicitly discussed. Selecting a smaller full model can offer a better trade-off than truncating a larger one. Likewise, local truncation provides modest computational savings, and peak-memory reductions depend strongly on batch size: approximately 25–27% at batch one, falling to 2–4% at batch 128 for Small/Base. These findings help define when stage removal is useful. The larger FLOP reductions obtained by shortening global S3 are reported separately and are not presented as measured speedups.

  • Purpose of the representation analyses. Proximity to the classifier can contribute to earlier class decodability, consistent with Alain and Bengio. We therefore emphasize the distinction between accessible class information and useful terminal computation. In the Nano allocation controls, models without S4 have stronger intermediate S3 readouts while remaining less accurate with their native classifier at comparable parameter budgets. Fixed-weight restrictions additionally reveal increased dependence on late S3 spatial exchanges after training without S4. These analyses document how the surviving hierarchy adapts and demonstrate why improved probe accuracy alone cannot establish that S4 is replaceable.

  • Lightweight terminal module. We have removed this module from the main contributions. The revised contribution rests on the controlled architectural comparisons and analyses of adaptation.

We hope these revisions clarify the scientific insight offered by the study and address your concerns about its novelty, interpretation, and practical scope.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.