- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes NewOrder, a structured pruning framework for Earth Observation Foundation Models (EOFMs) that progressively removes less important attention heads and MLP neurons during early downstream adaptation. The method combines output-sensitive scoring, progressive pruning, and representative calibration to account for changes in structural importance during training. Experiments across four foundation models and four downstream tasks show that, with approximately 50% encoder parameter compression, NewOrder achieves the best performance among the compared pruning methods in 14 out of 16 settings, while providing training and inference speedups.
Critical strengths
-
NewOrder performs structured pruning during early downstream adaptation, allowing subsequent training to operate directly on the compact model while also reducing deployment costs.
-
The paper systematically evaluates NewOrder across multiple EOFMs and downstream tasks, reporting actual inference latency, training-step latency, and peak memory consumption. The ablation studies also demonstrate the effectiveness of the main components.
Critical weaknesses
-
According to Appendix A.5, the baseline methods perform one-shot pruning before training, whereas NewOrder employs progressive pruning during early adaptation. Although the ablation studies demonstrate the effectiveness of individual components, comparing different pruning criteria under the same progressive pruning schedule would help better establish the independent contribution of output-sensitive scoring.
-
The main components of NewOrder build upon established techniques, including gradient-based saliency, progressive pruning, and representative calibration. In particular, Appendix D.1 shows a high overlap between the structures selected by NewOrder and Weighted Gradient. Although their combination achieves promising results, the paper could more clearly articulate its methodological contributions beyond existing structured pruning approaches.
Questions for authors
-
Can the authors compare existing pruning criteria under the same progressive pruning schedule to further isolate the contribution of output-sensitive scoring?
-
Given the high overlap between NewOrder and Weighted Gradient in structural selection, as well as existing research on progressive pruning, what is the key methodological novelty of NewOrder?
3 comments
Thank you for your responses. I maintain my score from the positive side.
Dear Reviewer cmaT,
Thank you so much for replying. Your comments were very helpful in improving the framing and clarifying the contribution of our paper!
Best,
#5165 Authors
Dear Reviewer cmaT,
Thank you for the thoughtful and constructive review. We appreciate your recognition of the benefits of early pruning, the measured efficiency gains, and the component ablations. We address your concerns below.
We are happy to see that the original scoring ablation has provided useful insights on the individual effectiveness of different components of NewOrder. Following your suggestion, we have extended the comparison to Magnitude, Activation, Weighted Gradient, and Taylor under the same progressive pruning schedule on HLS over three seeds.
From the table, we can see that NewOrder achieves the best performance. We also observe that most baselines improve over their one-shot counterparts after adopting progressive pruning, demonstrating that the effectiveness of progressive pruning is transferable to existing pruning methods. NewOrder's continued advantage under the shared schedule suggests that its full performance gains extend beyond the progressive schedule itself. The controlled scoring and calibration ablations in the original submission further identify the contributions of these components. We have added these results to Section 3.4.
We would like to first clarify that, as a unified pruning framework, NewOrder treats the selection of retained pretrained structures as part of downstream adaptation. Previous EOFM pruning approaches only partially address two challenges: structural rankings can change as the encoder and decoder adapt jointly, and rankings estimated from limited calibration inputs may underrepresent heterogeneous EO conditions. Moreover, loss-based saliency weights structural sensitivity by current prediction errors, which can make selection overly specific to the errors represented in the calibration set.
NewOrder addresses these challenges jointly. Output-sensitive scoring measures the current task response's dependence on each structure without loss-based error weighting. Representative calibration makes these estimates cover diverse geographic and land-cover conditions, while progressive re-estimation updates them throughout early adaptation. Together, these components allow structural selection to follow the evolving downstream task across representative EO inputs.
Within this framework, output-sensitive scoring makes a specific distinction from Weighted Gradient. For a structure g, the two objectives give Sout(g)=∣C∣1∑x∈C∑j∈gθj∂θj∂J(x;θ), and Sloss(g)=∣C∣1∑(x,y)∈C∑j∈gθj∂θj∂L(f(x;θ),y), respectively.
The supervised loss gradient weights output sensitivity according to the current prediction error. Our criterion directly measures how strongly the current task response depends on each structure.
Regarding Appendix D.1, the reported Jaccard overlap measures agreement in the identities of retained structures and counts each structure equally. Both criteria depend on output derivatives, making substantial agreement plausible, while the error weighting above can change the retained structures. Structures can nevertheless differ substantially in their influence on the model output. High overlap therefore leaves open whether the differing selections affect downstream performance, which the controlled comparisons in Fig. 3(a) directly evaluate (NewOrder's Avg mIoU of 89.25 compared to weighted gradient's 88.69).
We have clarified the rationale for the framework in Section 2.2 and expanded the structural-overlap discussion in Appendix D.
Happy to engage in further discussion!
Thank you again for the thoughtful review! We hope our responses address your concerns, and we would be happy to discuss further if you have any additional questions.