- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies structured LLM pruning when the effect of removing a component depends on which other components are removed. CoCurve builds a graph from single-unit forward ablations, representing individual pruning costs and interactions among attention heads and FFN groups. It then selects components jointly and chooses the interaction strength using teacher KL on separate calibration data. Experiments across LLMs and VLMs show improved quality retention over the evaluated selectors, including after common LoRA recovery and quantization. Physically sliced checkpoints also demonstrate inference throughput and memory improvements.
Critical strengths
- Broad, consistent evaluation. The study covers six LLMs, three VLMs, multiple pruning levels, recovery, and quantization. The quality gains therefore extend beyond one checkpoint or an unrecovered pruning result (Section 3; Appendices C–D).
- Controls isolate the practical contribution. Fixed-allocation and diagonal controls show gains from interaction-aware selection, while matched physical slicing separates retained quality from differences in memory or throughput (Table 7; supplementary Tables 18 and 30).
Critical weaknesses
- The named baselines are evaluated as modified selectors. Appendix C.3 removes FLAP's bias compensation, SlimGPT's reconstruction, and SlimLLM's regression. This is useful for isolating mask selection, but does not establish superiority over their complete published pipelines. Common LoRA recovery addresses a different comparison; the headline claims should preserve this distinction.
- The additional offline cost lacks a matched-compute comparison. Table 19 reports 17.4 hours for the 8B five-ratio sweep and 62.8 hours for 70B, including extensive candidate evaluation. The cost is disclosed, but the study does not establish whether comparable resources spent on alternative selection or recovery would retain the same quality ordering.
Questions for authors
- Are any existing results available with the baselines' native compensation/reconstruction enabled? Otherwise, can the claimed improvement be explicitly limited to the shared selector protocol?
- From the recorded runs, what are selection cost and quality for one requested pruning ratio, and how do these compare with the strongest baseline under the same device and resource accounting?
5 comments
Response to mpNv (2/3): Single-Target Cost and Graph Reuse
Part 2. Matched Compute (W2 / Q2)
Graph construction is an offline investment shared by multiple subsequent deployment versions. The resulting graph and pruned structures can also be delivered directly to users. Its return is a better structural starting point, reducing the recovery training needed to reach a given quality. We first show the demonstrated reuse scope and deployment costs in Part 2.1, then include amortized graph construction in the total cost to test the quality ordering when comparable resources go to alternative selection or recovery in Part 2.2.
Part 2.1. Cost and Reuse (Q2)
Takeaway. A base model's graph is reusable across ratios, and released selection results remain usable for reconstruction, adaptation, and low-bit deployment.
① Reuse is realized through ready-to-use graphs and structures. With the base model and structural units fixed, one graph supports searches at all pruning ratios. It occupies only 2.36 MB for 8B and under 15 MB for 70B. The reproduction package provides graphs for 9 models and 42 pruned structures, with exact CPU replay verified at all 42 operating points. For these base models, users can directly use the available graphs and selection results without rebuilding the graph:
- New pruning ratio: Reuse the graph and only search and score for that target. On an A100, searching 30%/50% structures for 8B takes 40.8/27.4 minutes.
- Target-ratio calibration coefficients available: Generate the target structure on CPU. For 8B at 20%/50%, this takes only 39/79 milliseconds.
- Released structure available: Skip search and compilation, proceeding directly to reconstruction or adaptation. Selection results remain usable for training-free reconstruction, LoRA recovery, and physically sliced low-bit deployment. See
Part 1andvvVx Part 4.
② Reuse costs correspond to actual quality gains. Separate cost runs compare complete preparation and quality at 50% pruning without LoRA, using A5000/A100 for 8B/12B, respectively. Δ denotes CoCurve + FFN Refinement relative to SlimGPT native.
| Model | CoCurve prep. (min) | SlimGPT prep. (min) | ΔC4 PPL (%) ↓ | ΔAvg12 (pp) ↑ |
|---|---|---|---|---|
| 8B | 51.6 | 48.7 | -40.0% | +0.65 |
| 12B | 54.3 | 50.0 | -9.3% | +4.58 |
8B requires only about 3 additional minutes of preparation, with 40.0% lower C4 and 0.65 percentage points higher Avg12. 12B requires about 4.4 additional minutes of preparation, with 9.3% lower C4 and 4.58 percentage points higher Avg12. The structural advantage produced by selection can thus reach users at subsequent preparation costs close to the native baseline's. The complete common-recovery table in Part 1.2 further shows that these selection results provide a better starting point for adaptation.
③ First-run single-target cost and amortization. For one requested pruning ratio, 8B at 50% on A5000 takes 21.85 hours for first-run selection (20.69 for construction and 1.16 for search/scoring), or 24.61 hours for the complete pipeline including slicing, FFN reconstruction, and one LoRA epoch. On the same device type, SlimGPT takes 0.81/2.95 hours for native preparation/the complete pipeline after the same one-epoch recovery. Both timings exclude evaluation. CoCurve's output has 6.3% lower C4 and 0.23 percentage points higher Avg12.
The 20.69 hours produce a shared graph: across 20%/30%/40%/50%, each ratio bears only 5.17 hours of construction, with its own search and recovery charged separately. The complete pipeline for a new target reusing the graph takes 3.92 hours (sum of measured stages). Construction time varies by device: 12.6/3.70/8.41 hours for 8B on RTX 6000 Ada/H100/A100. Table 19 in the submission (Table 22 in the revision) reports 17.4/62.8 hours for total selection across five ratios on 8B/70B, respectively. We next use complete resource comparisons to show the quality return from sharing this investment.
Response to mpNv (3/3): Quality–Cost Trade-offs Across Compute Budgets
Part 2.2. Quality–Cost Trade-off (W2 / Q2)
Takeaway. Including amortized construction, CoCurve achieves higher quality in less time. Its advantage also holds for complete native pipelines.
① Including amortized construction: better selection reduces recovery needs. To test the opportunity cost of additional selection, we use the same LoRA recipe on structures pruned under the paper's shared selection protocol, allowing LLM-Pruner more recovery compute. The table gives the full recovery curve for 8B at 50% on the same A5000 device type: CoCurve includes graph construction amortized across four ratios, target-specific search, slicing, and recovery. LLM-Pruner counts only training and saving.
| 8B, 50% pruning | Time (h) | C4 PPL ↓ | Avg12 (%) ↑ |
|---|---|---|---|
| CoCurve + 1 LoRA epoch | ≤8.70 | 33.89 | 45.15 |
| CoCurve + 4 LoRA epochs | ≤14.66 | 32.44 | 45.61 |
| LLM-Pruner + 1 LoRA epoch | 1.97 (training + save) | 37.14 | 44.57 |
| LLM-Pruner + 4 LoRA epochs | 7.89 (training + save) | 34.92 | 45.09 |
| LLM-Pruner + 8 LoRA epochs | 19.39 (training + save) | 39.21 | 45.13 |
| LLM-Pruner + 16 LoRA epochs | 33.98 (training + save) | 83.59 | 43.00 |
One CoCurve epoch takes less than half the time of the baseline's best downstream setting, reaching similar Avg12 with 13.6% lower C4. Four epochs are still faster than that setting and improve both metrics. Compared with the baseline's best measured value for each metric, four-epoch CoCurve has 0.48 percentage points higher Avg12 and 7.1% lower C4. Even charging all first-run graph construction, CoCurve's four-epoch total-time upper bound of 30.18 hours is below the baseline's 33.98 hours for training and saving alone at sixteen epochs. Thus, the offline investment both reduces recovery training needed for the same quality and is amortizable across pruning budgets.
② Complete native pipelines: the advantage persists at similar total time.
i) Shared four-epoch recovery. We keep native selection and reconstruction and apply four LoRA epochs with the same recipe. In this A5000 matched-compute comparison, CoCurve sums measured search, slicing, and recovery stages after graph reuse. Baseline times include full native preparation and training. Evaluation is excluded.
| 8B, 50% pruning | Time (h) | C4 PPL ↓ | Avg12 (%) ↑ |
|---|---|---|---|
| CoCurve + FFN Refinement | 9.71 | 23.00 | 48.16 |
| SlimGPT native | 9.37 | 24.09 | 48.11 |
| SlimLLM native | 8.17 | 27.72 | 47.83 |
| FLAP native | 9.23 | 28.39 | 48.02 |
Within roughly 8–10 hours, CoCurve achieves the lowest C4 (4.5–19.0% lower) and the highest Avg12. Reconstruction and additional LoRA both optimize within the retained structure, so better structure selection still affects the final recoverable quality, consistent with Part 1.2.
ii) Recovery budget sweep. On A100, CoCurve and LLM-Pruner reuse released structures. The three native baselines retain their selection and reconstruction. Time includes loading, slicing, and recovery. Each cell lists time (h)/C4↓/Avg12↑, excluding evaluation time. CoCurve’s four-epoch Avg12 is from a separate A100 run; time and C4 are from the original run.
| 8B, 50% pruning | 1 epoch | 2 epochs | 4 epochs | 8 epochs |
|---|---|---|---|---|
| CoCurve + FFN Refinement | 1.68 / 23.08 / 48.19 | 2.63 / 22.82 / 48.54 | 4.54 / 23.18 / 48.71 | 9.05 / 26.38 / 48.21 |
| SlimGPT native | 1.81 / 24.37 / 47.67 | 3.08 / 23.89 / 47.42 | 5.24 / 24.14 / 48.00 | 11.02 / 27.68 / 47.45 |
| SlimLLM native | 1.20 / 27.67 / 48.21 | 2.29 / 27.20 / 48.43 | 4.47 / 27.72 / 48.64 | 9.04 / 34.19 / 47.50 |
| FLAP native | 1.26 / 29.15 / 46.90 | 2.47 / 28.34 / 47.72 | 4.86 / 28.33 / 48.14 | 9.67 / 31.65 / 48.07 |
| LLM-Pruner | 0.98 / 37.13 / 44.60 | 1.92 / 35.53 / 44.81 | 3.83 / 34.92 / 45.09 | 7.68 / 39.23 / 44.89 |
CoCurve has the lowest C4 at every dose and the highest Avg12 at two, four, and eight epochs. Its two-epoch C4 (22.82) beats every baseline setting in the sweep. Against SlimGPT, one epoch takes 1.68 vs. 1.81 h (C4 −5.3%, Avg12 +0.51 pp), and four take 13.4% less time (C4 −4.0%, Avg12 +0.71 pp).
③ Alternative selection: the advantage persists with equal tuning opportunities. Beyond recovery, we give the interaction-aware selector SOSP-I the same calibration text and candidate limit to test whether selection tuning closes the gap. CoCurve still leads by 0.99–4.08 percentage points at all four 8B ratios from 20–50%. See vvVx Part 2.2.
Graph construction costs are shared across targets, while structural quality gains remain in each deployed model. These comparisons show that additional selection compute yields better structures, reduces reliance on recovery training, and delivers quality gains in complete native pipelines. The supplied graphs and structures make these returns directly reusable by subsequent users.
Thank you again for recognizing the practical contribution and for the suggestions that have helped us strengthen the end-to-end and resource comparisons.
Response to mpNv (1/3): Native Pipelines and Shared Recovery
Thank you for your constructive feedback and for recognizing our consistent empirical gains, controlled analyses, and practical deployment value. We address complete native pipelines in
Part 1and single-target cost and matched-compute comparisons inPart 2.Part 1. Native Pipelines (W1 / Q1)
We have added complete pipelines with FLAP's, SlimGPT's, and SlimLLM's native compensation/reconstruction enabled. The paper's shared selector protocol isolates structure selection itself and establishes CoCurve's selection advantage (best PPL in 38/45 comparisons across five models; highest Avg12 in 8/8 configurations on 8B/24B). We first directly compare complete native pipelines in
Part 1.1, then test whether the quality advantage persists when every pipeline receives equal lightweight adaptation inPart 1.2.Part 1.1. Full Pipelines
Specifically, we retain each baseline's native selection and recovery: FLAP's channel allocation and bias compensation, SlimGPT's sequential selection and OBS reconstruction, and SlimLLM's inter-layer allocation, channel selection, and local regression. We implement CoCurve's complete pipeline with two existing reconstruction approaches, comparing each against the three native pipelines. CoCurve + OBS fixes the original mask and uses OBS to compensate retained weights. CoCurve + FFN Refinement preserves CoCurve's selected attention heads and per-layer quotas, reusing SlimGPT's channel selection and reconstruction within FFNs. All pipelines are evaluated after selection and recovery.
The table reports Avg12 (%) for three models. Each cell lists 30%/40%/50% pruning. No pipeline uses LoRA.
Against the strongest native baseline in each configuration, CoCurve + OBS achieves higher Avg12 in 8/9 configurations, while CoCurve + FFN Refinement achieves higher Avg12 in 9/9 and the lowest C4 in 8/9. CoCurve's structure-selection advantage carries through to complete pipelines across all three models.
Part 1.2. Shared LoRA
The paper's common LoRA comparison tests structural recovery potential under the same budget (Tables 3, 16). We further start from each complete native pipeline's output and apply the same one-epoch Alpaca LoRA (rank 16), testing whether CoCurve's structural advantage persists when the baselines also receive adaptation. Each cell lists 30%/40%/50%, with C4↓/Avg12↑ at each ratio.
Across six configurations, the CoCurve combination achieves the lowest C4 in five and the highest Avg12 in four. Complete native results directly establish end-to-end competitiveness. Common recovery further shows that the better structural starting point remains valuable during subsequent adaptation. The revision will include native results and specify which conclusions the shared selection protocol, complete native pipelines, and common recovery each support in the headline claims.