- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper tries to answer an interesting question: Do Hierarchical Vision Backbones Need a Fourth Stage? It conducts extensive experiments across different downstream tasks, model families, and model scales.
Critical strengths
- The research question sounds interesting.
- The paper provides comprehensive and detailed experiments and makes a substantial effort to discuss the findings thoroughly.
Critical weaknesses
The only but fundamental concern is that all the experiments in this paper can be explained by a familiar relationship between model capacity and data demands, rather than establish the architectural value or necessity of the fourth stage itself. Capacity redundancy is already widely understood. Observing it does not constitute a sufficiently novel or valuable work. The experiments do not sufficiently rule out this simpler explanation. Therefore, it remains unclear whether the results reflect a distinct architectural role of the fourth stage itself.
For example, the paper’s central observation is that removing the fourth stage has a smaller impact on ImageNet-1K performance as Swin and ConvNeXt become larger. However, this observation is fully consistent with a capacity-based explanation: removing S4 from narrow models eliminates a substantial portion of their capacity, leading to a drop of 2–3 percentage points, whereas Small/Base models lose only 0.13–0.53 points because the preceding stages may retain sufficient capacity for the task. These results therefore do not distinguish the contribution of the fourth-stage architecture from the effect of overall model capacity.
After reviewing all the experimental designs and results, the reviewer could not identify enough evidence that goes beyond the "capacity redundancy" explanation. The concern is whether this paper provides new insight into the fourth stage itself. However, the reviewer is open to discuss about it.
In addition, the reviewer finds the paper's organization and presentation confusing. The Appendix contains 18 sections and a large number of experimental results, while the main paper occupies only eight and a half pages and does not appear to be carefully formatted.
Questions for authors
Please address the concern raised in the weaknesses and provide additional experiments to support the response. Please discuss the function of fourth stage and analyze why the fourth stage would be not necessary in some cases.
Thank you for recognizing the systematic investigation and the insights provided by the experiments. We have clarified the paper’s objective: explaining when S4 contributes useful computation, which upstream changes can substitute for it, and how the remaining hierarchy adapts.
Contribution beyond pruning results (W1, Q2). Preserving accuracy with fewer parameters is established, and HiViT demonstrates competitive three-stage architectures. Our contribution concerns the conditions behind that possibility. We cross local/global S3 mixing with S4 presence at fixed S3 depth and width, retaining the final downsampling. These controls show that upstream spatial computation changes the benefit of S4. Shortening global S3 to 14 blocks then allows Swin-S to reach 83.10% versus 83.04% for its full local reference, with approximately 43% fewer parameters and 21% fewer FLOPs in the reported runs. A deeper local alternative remains less accurate despite using more resources. The tested ConvNeXt kernel enlargements do not reproduce this recovery. These results identify architectural conditions and limits beyond a catalogue of compression trade-offs.
Label noise and ReaL (W1, Q1). Training-label noise and evaluation-label errors must be distinguished. Northcutt et al. examine evaluation annotations, whereas Yun et al. also address training supervision. ReaL revises evaluation labels without correcting our training labels. Nevertheless, it directly informs the proposed ranking reversal: the full model remains more accurate in all ten local full/truncated pairs under ReaL, although the truncation penalty decreases. Thus, truncated models do not overtake their full counterparts under reassessed labels. The cited pruning studies also support a conditional interpretation: Jin et al. distinguish training and regularization effects, while He et al. show that pruning can mitigate or aggravate overfitting depending on sparsity. These findings do not establish a general prediction for S4 removal. Our results constrain the proposed explanation without isolating sensitivity to training-label noise.
Interpretation of the representation measure (W2). The probe measures class accessibility under a specified readout, not representation quality generally. Top-1 ignores confidence and margins, and probe fitting and evaluation remain label-dependent. For depth discretization, every S3 block is evaluated, with identical block positions within each local full/truncated pair. Equation (3) summarizes the profiles through piecewise-linear interpolation after subtracting the input difference; it is a descriptive statistic, not a discretization-invariant quantity. Crucially, the Nano allocation controls show stronger intermediate probe accuracy alongside worse native accuracy at comparable parameter budgets. Fixed-weight spatial restrictions separately examine functional dependence. Together, these analyses distinguish accessible information from the computations supporting predictions.
Relation to structured pruning. The revised related work distinguishes criteria using learned weights alone from those requiring activations, gradients, or evaluation data. Weight-only criteria still inherit dependence on training data. Our removal target is prescribed by the architectural question: independent training tests what the modified architecture can learn, whereas fixed-weight interventions examine reliance in an existing model.
Variability and reproducibility. The revised manuscript reports three training runs per local full/truncated architecture except standard Tiny. Global and shortened variants are identified as individual runs. Probe intervals describe image-level uncertainty conditional on the checkpoints, separately from training variability. We will explicitly state code availability in the reproducibility statement.
Tiny profiles and presentation. Standard Tiny and Tiny-C64 differ in depth and allocation, so their profiles cannot be interpreted as a width-only effect. Proximity to the classifier may also contribute to earlier decodability. The revised interpretation emphasizes these distinctions alongside functional dependence. The original overview figure has been replaced, with explicit labels distinguishing full/truncated models and allocation controls.
We hope these revisions clarify the explanatory contribution and the scope of the architectural and representation evidence.