- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DSASP combines Taylor and empirical-Fisher importance scores with cross-sample magnitude dispersion and ranking consistency, then optimizes their weights using TPE with adaptive search boundaries. Experiments on LLaMA-7B and Vicuna-7B report improved perplexity and mean zero-shot accuracy at two pruning ratios. The intended contribution is that stability information and adaptive configuration search jointly improve structural selection.
Critical strengths
- Distinguishing average importance from its consistency across calibration samples is a reasonable direction for improving pruning under limited data.
- Consistent experimental improvements. Tables 2–5 show lower perplexity in all four settings. Recalculating the seven-task averages also confirms the abstract’s claim of the highest mean accuracy in all four settings.
Critical weaknesses
- The methodological novelty beyond existing adaptive pruning frameworks is insufficiently established. AdaPruner[1] already formulates calibration and importance-estimation configuration as a Bayesian optimization problem, while FLAP[2] already exploits cross-sample activation fluctuations for structured pruning.
- The experimental results does not yet establish the claimed complementary contributions. Tables 7–8 remove RC and dispersion together, so neither their individual value nor their complementarity is demonstrated; the combined stability gain on Vicuna is only 0.11 percentage points, without repeated-run uncertainty. The ablations also use a different search budget and omit their pruning ratios. Factorial ablations with matched settings, multiple calibration/search seeds, and convergence-versus-cost curves are needed.
- The motivation for dynamic search conflates adapting exploration with enlarging the feasible configuration space. The introduction presents fixed search boundaries as a limitation, yet DSASP starts with the entire domain , and Equation (9) confines every subsequent region to that same domain. Consequently, expansion only revisits configurations already admissible to the original fixed-domain optimizer; its potential benefit is a better allocation of evaluations, which requires evidence from matched-budget convergence comparisons. Adaptive search regions also have substantial precedent in Bayesian optimization, including TuRBO[3].
References
[1] Kong et al. Sample-aware Adaptive Structured Pruning for Large Language Models
[2] An et al. Fluctuation-Based Adaptive Structured Pruning for Large Language Models
[3] Eriksson et al. Scalable Global Optimization via Local Bayesian Optimization
Questions for authors
- What are the sample identifiers and splits for calibration selection, proxy optimization, and final perplexity evaluation? Are they strictly disjoint?
- What pruning ratios were used in Tables 7–8, and what happens when RC and dispersion are ablated separately?
4 comments
Comment 4. What are the sample identifiers and splits for calibration selection, proxy optimization, and final perplexity evaluation? Are they strictly disjoint?
Response. Thank you for requesting a clearer account of the three stages. The stages are procedurally separated and serve different roles.
Evidence. First, DSASP selects 10 calibration samples from 300 BookCorpus candidates and uses only these samples to compute the sample-wise structural statistics. Second, TPE evaluates each candidate weight configuration on a fixed 16-sample WikiText-2 proxy. This small proxy is used only as a low-cost surrogate for comparing candidate masks; WikiText-2 is not required by the method, and another small proxy corpus could be used. Third, after the search terminates, the selected weights and pruning mask are fixed, and the model is evaluated using the 128-sample WikiText-2 and PTB protocols and seven zero-shot tasks. PTB and all zero-shot tasks are never used by the optimizer. This follows the AdaPruner-style calibration-proxy-evaluation pipeline while keeping the final downstream evaluation outside the search loop.
Comment 5. What pruning ratios were used in Tables 7–8, and what happens when RC and dispersion are ablated separately?
Response. Thank you for asking us to make the ablation settings explicit.
Evidence. Table 7 reports LLaMA-7B at 40% pruning, while Table 8 reports Vicuna-7B at 20% pruning. We have added these ratios to the table captions. The newly added component-wise rows in Tables 7 and 8
Revision place. The captions of Tables 7 and 8 in Section 4.5 now include the corresponding model and pruning ratio.
Response to Author Rebuttal
Thank you for the clarifications and additional ablations. The separate RC/Var ablations, specified pruning ratios, and revised explanation of dynamic search address part of my concerns. However, procedural separation does not establish that the WikiText-2 proxy and final evaluation samples are disjoint. The small Vicuna gain also remains difficult to assess without repeated-run uncertainty. While the positioning is clearer, I remain unconvinced that the methodological advance is sufficiently substantial. I therefore maintain my original score.
Response. Thank you for your follow-up. We address the remaining concerns below.
Our previous response distinguished the roles of calibration, configuration search, and final evaluation but did not directly state the relationship between the corresponding WikiText-2 samples. The 16 WikiText-2 examples used by TPE are contained in the 128-example WikiText-2 pool used for PPL reporting. Following AdaPruner’s evaluation-guided Bayesian optimization procedure , DSASP uses perplexity on this small evaluation set as a low-cost signal for comparing candidate pruning configurations.
BookCorpus supplies the calibration statistics, while PTB and all seven zero-shot tasks are never accessed by TPE or used for configuration selection. Across all four model–pruning-ratio settings, DSASP achieves the lowest PTB perplexity and the highest mean zero-shot accuracy among the evaluated pruning methods. These results provide performance evidence beyond the WikiText-2 signal used during configuration search.
The repeated-run question concerns variation across calibration and search seeds. Tables 7–8 serve as controlled component comparisons: within each setting, all variants use the same model, pruning ratio, calibration data, proxy set, search budget, seed, and evaluation protocol. On Vicuna-7B, the mean accuracies for full, w/o RC, w/o Var, and w/o both are 59.43, 59.19, 59.12, and 59.32, respectively. On LLaMA-7B, the corresponding values are 49.69, 48.50, 48.52, and 48.47. The 0.11-point value is one full-versus-w/o-both comparison, while removing RC or Importance Variance produces the same directional effect in both settings. The fixed-region results, 58.34 on Vicuna-7B and 48.69 on LLaMA-7B, further isolate the contribution of dynamic search. Because the remaining weights are re-optimized and each variant produces a discrete pruning mask, the component effects are not expected to be additive. The evidentiary role of these ablations is to isolate the observed component effects under matched conditions; the four main settings and the out-of-proxy benchmarks provide the broader performance evidence.
DSASP’s contribution is the coupled pruning formulation defined by Equations (8)–(12). Equation (8) integrates Taylor and empirical-Fisher structural evidence with magnitude dispersion and relative-rank consistency. Equations (9)–(12) allocate a finite TPE evaluation budget between local refinement and broader exploration within the fixed global domain. AdaPruner searches calibration samples and importance-estimation configurations, FLAP uses activation fluctuation for sparsity allocation, and TuRBO addresses general Gaussian-process trust-region optimization. DSASP instead models two forms of stability in sample-wise gradient-based structural evidence and jointly optimizes their fusion through feedback-controlled TPE search.
The complete source code and experimental scripts are included in the supplementary attachment for direct verification and reproduction of the reported experiments.
Comment 1. What is the methodological novelty of DSASP compared with AdaPruner and FLAP?
Response. Thank you for encouraging us to sharpen the distinction from AdaPruner and FLAP.
Evidence. The submitted manuscript already describes the relevant components separately: Section 2.1 identifies FLAP as an activation-fluctuation method for sparsity allocation, Section 2.3 describes AdaPruner as searching calibration data and importance-estimation configurations, and Sections 3.2-3.3 define DSASP’s stability-aware score and dynamic region. The distinction is that DSASP combines two mechanisms not used jointly by these methods. First, it augments Taylor and empirical-Fisher contribution evidence with both magnitude dispersion and relative-rank consistency computed from sample-wise structural evidence. Second, it reallocates the finite TPE search budget by shifting, contracting, and expanding the admissible local region according to optimization feedback. AdaPruner searches different objects within fixed global bounds, while FLAP uses activation fluctuation rather than ranking consistency of gradient-based structural evidence.
Comment 2. Do RC and Importance Variance make separate and complementary contributions to DSASP?
Response. Thank you for requesting component-wise evidence for the two stability statistics.
Evidence. We added separate removal ablations under the same model, pruning ratio, search budget, and evaluation protocol as Tables 7 and 8. Full DSASP achieves 49.69% on LLaMA-7B at 40% pruning, compared with 48.50% without RC, 48.52% without Var, and 48.47% without both. On Vicuna-7B at 20% pruning, the corresponding values are 59.43%, 59.19%, 59.12%, and 59.32%. Full DSASP is best in both settings, and removing either statistic reduces mean accuracy. The effects need not be additive because the remaining weights are re-optimized and each variant induces a discrete pruning mask.
\hline \mathbf{Setting} & \mathbf{Full} & \mathbf{w/o\\ RC} & \mathbf{w/o\\ Var} & \mathbf{w/o\\ RC\\ and\\ Var} \\\\ \hline \text{LLaMA-7B 40\\%} & 49.69 & 48.50 & 48.52 & 48.47 \\\\ \text{Vicuna-7B 20\\%} & 59.43 & 59.19 & 59.12 & 59.32 \\\\ \hline \end{array}$$ Revision place. We have added the norc and novar variants to Tables 7 and 8 and expanded the component-wise analysis at the end of Section 4.5. Comment 3. If the initial search region already covers $[0,1]^4$, how does the dynamic search region improve optimization under a finite evaluation budget? Response. Thank you for asking us to clarify the role of expansion after the initial global region and for pointing out TuRBO as a relevant precedent. Evidence. TuRBO performs Bayesian optimization using local Gaussian-process trust regions, expanding a region after consecutive successes and shrinking it after consecutive failures (Eriksson et al., 2019). DSASP differs in its search variables, optimization framework, and region-update rule. DSASP uses TPE to search the four fusion weights of the Taylor, empirical-Fisher, RC, and Importance Variance terms within the fixed global domain $[0,1]^4$. As specified in Section 3.3, Eq. (9) initializes the admissible region over the entire domain and constrains every subsequent region to remain within these global bounds. When an improvement is found, Eqs. (10)–(11) shift the center toward the improving configuration and contract the radius, increasing the search resolution around a promising region. When progress stalls, Eq. (12) re-expands the previously contracted region, thereby broadening exploration within the global domain. Expansion therefore does not create configurations outside the original feasible space. Its role is to adapt the focus and resolution of the search under a finite evaluation budget: DSASP begins with full-domain exploration, concentrates evaluations around promising configurations, and restores broader exploration when the concentrated search no longer improves. Under the matched search budget reported in Tables 7–8, the dynamic region improves mean zero-shot accuracy from 48.69% to 49.69% on LLaMA-7B at 40% pruning and from 58.34% to 59.43% on Vicuna-7B at 20% pruning. Revision place. We have added TuRBO to Section 2.3 and the reference list, and clarified that DSASP dynamically adjusts the resolution and allocation of search within the fixed global domain rather than enlarging the feasible configuration space.