- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces ReTaPER, a task-aware structured pruning and recovery framework for large language models (LLMs). The study reveals that the block-wise importance of a model can be decomposed into a general component and a task-specific residual component, with the latter residing approximately within a low-rank space. Using the various tasks within the MMLU benchmark as a foundation, ReTaPER isolates these task-specific importance residuals and applies singular value decomposition (SVD). For a new task, the importance residuals for all blocks can be approximated by evaluating only a small number of "anchor blocks." Experiments conducted on Llama-3.1-8B and Qwen3-8B models across nine downstream tasks demonstrate that ReTaPER outperforms existing pruning baseline methods on average.
Critical strengths
(1) Decomposing "module importance" into a "general" component and a "low-rank, task-specific" component offers a valuable perspective for "task-aware structured pruning." The motivating experiments are quite thorough, demonstrating that the observed phenomenon—where low-rank importance is linked to specific tasks—is a general property rather than merely a coincidental correlation between matrix elements.
(2) The paper proposes a practical framework that integrates importance residual reconstruction, sparse anchor block prediction, and dynamic LoRA rank allocation into a coherent pipeline. Across two different backbone models, this approach consistently outperforms the baselines.
Critical weaknesses
(1) Although the paper includes several representative baseline methods, it fails to incorporate several closely related "task-aware block-pruning" approaches (such as TaBP and TELL-TALE, discussed in Section 2) into the primary comparative experiments. This weakens the paper's claim that its proposed method outperforms existing "task-aware structured pruning" techniques.
(2) In Table 1, the variance in accuracy across certain tasks—notably on the FIQA-SA dataset, where the performance gap compared to IG-Pruning is substantial—undermines the persuasiveness of the reported average accuracy metrics. Furthermore, the evaluation is limited to models with approximately 8B parameters; the lack of testing on larger or different model architectures limits the demonstrated generalizability of the approach.
(3) The paper presents the use of a "residual signal for dynamic LoRA rank allocation" as a key contribution. However, it fails to provide a comparison against other LoRA methods that employ non-uniform rank allocation.
(4) Regarding calibration efficiency, the paper only evaluates the speed of calibration for new tasks. For offline calibration methods, however, ReTaPER’s initial calibration step using the MMLU dataset is significantly more time-consuming—even if this step is only performed once.
(5 )What specific calibration data was used for the baselines? The current experimental design does not rule out the potential influence of the MMLU calibration data used in ReTaPER. It remains unclear whether the performance gains observed with ReTaPER stem from the method itself or from the additional, preliminary calibration performed on the MMLU dataset.
Questions for authors
Refer to the Critical Weaknesses.
7 comments
Response to Reviewer kLwB 2/6
W2/Q2 (task sensitivity): Is the average-accuracy advantage driven by FIQA-SA?
We added a complete leave-one-task-out analysis in Appendix E.2. For each omitted task, we recompute every method’s unweighted mean over the remaining eight tasks and reselect the strongest pruned baseline. The analysis uses Table 1 without retraining. Below, accuracy is in percent; Delta is ReTaPER minus the best baseline in percentage points.
Llama-3.1-8B
| Omitted task | Best baseline (No FT) | ReTaPER (No FT) | Delta | Best baseline (FT) | ReTaPER (FT) | Delta |
|---|---|---|---|---|---|---|
| None (9 tasks) | 53.48 | 56.15 | +2.67 | 60.70 | 66.97 | +6.27 |
| PubMedQA | 51.81 | 53.92 | +2.10 | 59.62 | 66.64 | +7.03 |
| MedMCQA | 56.42 | 60.19 | +3.77 | 63.76 | 68.46 | +4.70 |
| LICQ | 54.72 | 55.95 | +1.23 | 62.33 | 66.95 | +4.61 |
| FIQA-SA | 54.34 | 51.95 | -2.39 | 57.32 | 63.86 | +6.54 |
| PIQA | 51.01 | 55.11 | +4.10 | 59.47 | 66.20 | +6.73 |
| HellaSwag | 51.99 | 56.44 | +4.45 | 61.08 | 67.85 | +6.76 |
| WinoGrande | 52.53 | 55.38 | +2.86 | 60.18 | 66.70 | +6.52 |
| ARC-E | 52.64 | 57.66 | +5.02 | 60.60 | 66.03 | +5.43 |
| ARC-C | 55.86 | 58.75 | +2.90 | 63.24 | 70.06 | +6.82 |
Qwen3-8B
| Omitted task | Best baseline (No FT) | ReTaPER (No FT) | Delta | Best baseline (FT) | ReTaPER (FT) | Delta |
|---|---|---|---|---|---|---|
| None (9 tasks) | 50.75 | 54.60 | +3.85 | 58.88 | 64.53 | +5.65 |
| PubMedQA | 50.00 | 53.40 | +3.41 | 59.32 | 65.08 | +5.76 |
| MedMCQA | 53.75 | 57.60 | +3.86 | 61.88 | 65.94 | +4.06 |
| LICQ | 51.48 | 54.66 | +3.18 | 58.84 | 63.08 | +4.24 |
| FIQA-SA | 51.38 | 53.13 | +1.75 | 54.79 | 60.81 | +6.03 |
| PIQA | 48.07 | 52.78 | +4.71 | 57.11 | 63.80 | +6.68 |
| HellaSwag | 49.88 | 55.03 | +5.15 | 59.69 | 66.11 | +6.43 |
| WinoGrande | 49.76 | 53.06 | +3.30 | 59.56 | 64.19 | +4.64 |
| ARC-E | 49.66 | 55.53 | +5.87 | 59.04 | 64.43 | +5.39 |
| ARC-C | 52.80 | 56.25 | +3.45 | 61.23 | 67.37 | +6.14 |
Without fine-tuning, IG-Pruning is the strongest baseline in every subset. With fine-tuning, it is GradPruner on Llama except when omitting MedMCQA or LICQ (LLM-Streamline), and LLM-Streamline on Qwen except when omitting WinoGrande (GradPruner).
ReTaPER leads in 35 of 36 comparisons (97.22%; two backbones × two fine-tuning settings × nine omitted tasks). We acknowledge that the Llama pruning-only advantage is sensitive to FIQA-SA: removing it reverses the gap to −2.39 points. However, after recovery, every exclusion retains a positive gap, with minimum gains of 4.61 points on Llama and 4.06 on Qwen. These results support robustness across the tested task subsets while recognizing task-dependent performance. The overlapping subsets are sensitivity checks, not independent statistical trials.
Response to Reviewer kLwB 3/6
W2/Q2 (model generalization): Evaluation is limited to approximately 8B-parameter models, leaving generalization to larger or different architectures unclear.
We have extended evaluation to Qwen3-30B-A3B, which has a larger total parameter count and a mixture-of-experts architecture, complementing the two 8B dense backbones. The complete task-level results are shown below. Accuracy is in percent, and Average is the unweighted nine-task mean. Base Model and Tuned Base Model are unpruned references.
Qwen3-30B-A3B, without fine-tuning.
| Task | Base Model | TaBP | IG-Pruning | ReTaPER |
|---|---|---|---|---|
| PubMedQA | 78.40 | 74.00 | 68.20 | 75.80 |
| MedMCQA | 52.14 | 27.25 | 30.76 | 28.02 |
| LICQ | 48.63 | 44.25 | 45.08 | 58.25 |
| FIQA-SA | 79.91 | 57.69 | 50.31 | 88.32 |
| PIQA | 80.47 | 72.74 | 73.65 | 64.29 |
| HellaSwag | 77.75 | 52.34 | 65.63 | 54.26 |
| WinoGrande | 70.72 | 60.54 | 61.90 | 62.51 |
| ARC-E | 79.08 | 40.49 | 60.18 | 43.98 |
| ARC-C | 55.89 | 34.64 | 35.08 | 37.72 |
| Average | 69.22 | 51.55 | 54.53 | 57.02 |
Qwen3-30B-A3B, with fine-tuning.
| Task | Tuned Base Model | GradPruner | ReTaPER |
|---|---|---|---|
| PubMedQA | 76.60 | 58.40 | 65.80 |
| MedMCQA | 62.31 | 46.54 | 57.36 |
| LICQ | 74.91 | 61.41 | 77.50 |
| FIQA-SA | 88.36 | 90.53 | 93.97 |
| PIQA | 84.76 | 74.89 | 72.83 |
| HellaSwag | 78.02 | 53.12 | 55.79 |
| WinoGrande | 71.73 | 59.76 | 69.22 |
| ARC-E | 85.94 | 63.37 | 68.41 |
| ARC-C | 63.27 | 42.93 | 44.78 |
| Average | 76.21 | 61.22 | 67.30 |
Without fine-tuning, ReTaPER reaches 57.02 average accuracy, exceeding IG-Pruning by 2.49 points and TaBP by 5.47 points. After fine-tuning, it reaches 67.30, exceeding GradPruner by 6.08 points and leading on eight of nine tasks. Recovery raises ReTaPER's average from 57.02 to 67.30, including gains of 29.34 points on MedMCQA and 24.43 points on ARC-E.
These results extend the evidence to the tested larger MoE backbone. The gains remain task-dependent: IG-Pruning is stronger on PIQA, HellaSwag, and ARC-E without fine-tuning, and GradPruner remains stronger on PIQA after fine-tuning. The new results and analysis are included in Appendix F.1 (Table 12), with a corresponding main-text summary.
Response to Reviewer kLwB 1/6
We thank the reviewer for the constructive feedback. We have revised the manuscript accordingly, with changes highlighted in blue in the updated manuscript. Detailed point-by-point responses are provided below.
W1/Q1: The main comparison lacks closely related task-aware block-pruning baselines, especially TaBP and TELL-TALE.
We have added TaBP to the main accuracy and efficiency comparisons. ReTaPER exceeds TaBP by 4.63 and 8.93 percentage points in nine-task average accuracy on Llama-3.1-8B and Qwen3-8B, respectively, without post-pruning fine-tuning.
TaBP derives task-specific pruning plans from output-distribution shifts. Its calibration uses up to 2,500 labeled examples per downstream task, with the data settings now specified in Appendix D.2. The following pruning-only comparisons use the main experiment's 25% target parameter-pruning ratio. Accuracy is reported in percent; Average is the unweighted mean over the nine tasks.
Llama-3.1-8B, without fine-tuning.
Qwen3-8B, without fine-tuning.
ReTaPER retains the highest average accuracy in these comparisons after adding TaBP. The task-level results also show that the advantage is not uniform: for example, IG-Pruning remains stronger on PIQA and ARC-E on both backbones. These results strengthen the comparison against task-aware pruning, while our claim remains an average-accuracy advantage over the evaluated baselines. Tables 1 and 3 and the corresponding experimental descriptions have been updated.