acceptodds
Under review as a conference paper at ICLR 2027

Task Residuals Are Low-Rank: Efficient Pruning and Recovery of Large Language Models

Abstract

Structured pruning reduces the inference cost of large language models by removing complete architectural units. However, existing methods either rely on task-agnostic importance estimates that overlook task-specific structural requirements or repeatedly evaluate all candidate structural blocks for each downstream task, resulting in substantial adaptation overhead. Through a systematic block-wise masking study, we empirically identify a previously overlooked structure in block importance: it can be decomposed into a generic component shared across tasks and task-specific residuals concentrated in a compact low-rank subspace. Building on this finding, we propose ReTaPER, a residual subspace-guided framework for task-aware pruning with efficient recovery. ReTaPER first learns a low-rank residual basis from block-masking responses. For a new task, it probes only a small set of informative anchor blocks, reconstructs the complete task residual within the learned subspace, and combines it with generic importance to derive a task-specific pruning order. After pruning, the reconstructed residual further guides the allocation of a fixed LoRA rank budget, assigning greater adaptation capacity to retained blocks that are more sensitive to the target task. Extensive experiments across nine tasks demonstrate that ReTaPER achieves state-of-the-art pruning performance both with and without recovery. Code is available at https://anonymous.4open.science/r/ReTaPER-ICLR2027-4DD3/.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Model Compression, Structural Priors

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_kLwBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ReTaPER, a task-aware structured pruning and recovery framework for large language models (LLMs). The study reveals that the block-wise importance of a model can be decomposed into a general component and a task-specific residual component, with the latter residing approximately within a low-rank space. Using the various tasks within the MMLU benchmark as a foundation, ReTaPER isolates these task-specific importance residuals and applies singular value decomposition (SVD). For a new task, the importance residuals for all blocks can be approximated by evaluating only a small number of "anchor blocks." Experiments conducted on Llama-3.1-8B and Qwen3-8B models across nine downstream tasks demonstrate that ReTaPER outperforms existing pruning baseline methods on average.

Critical strengths

(1) Decomposing "module importance" into a "general" component and a "low-rank, task-specific" component offers a valuable perspective for "task-aware structured pruning." The motivating experiments are quite thorough, demonstrating that the observed phenomenon—where low-rank importance is linked to specific tasks—is a general property rather than merely a coincidental correlation between matrix elements.

(2) The paper proposes a practical framework that integrates importance residual reconstruction, sparse anchor block prediction, and dynamic LoRA rank allocation into a coherent pipeline. Across two different backbone models, this approach consistently outperforms the baselines.

Critical weaknesses

(1) Although the paper includes several representative baseline methods, it fails to incorporate several closely related "task-aware block-pruning" approaches (such as TaBP and TELL-TALE, discussed in Section 2) into the primary comparative experiments. This weakens the paper's claim that its proposed method outperforms existing "task-aware structured pruning" techniques.

(2) In Table 1, the variance in accuracy across certain tasks—notably on the FIQA-SA dataset, where the performance gap compared to IG-Pruning is substantial—undermines the persuasiveness of the reported average accuracy metrics. Furthermore, the evaluation is limited to models with approximately 8B parameters; the lack of testing on larger or different model architectures limits the demonstrated generalizability of the approach.

(3) The paper presents the use of a "residual signal for dynamic LoRA rank allocation" as a key contribution. However, it fails to provide a comparison against other LoRA methods that employ non-uniform rank allocation.

(4) Regarding calibration efficiency, the paper only evaluates the speed of calibration for new tasks. For offline calibration methods, however, ReTaPER’s initial calibration step using the MMLU dataset is significantly more time-consuming—even if this step is only performed once.

(5 )What specific calibration data was used for the baselines? The current experimental design does not rule out the potential influence of the MMLU calibration data used in ReTaPER. It remains unclear whether the performance gains observed with ReTaPER stem from the method itself or from the additional, preliminary calibration performed on the MMLU dataset.

Questions for authors

Refer to the Critical Weaknesses.

7 comments
AuthorsOpenReview ↗

Response to Reviewer kLwB 1/6

We thank the reviewer for the constructive feedback. We have revised the manuscript accordingly, with changes highlighted in blue in the updated manuscript. Detailed point-by-point responses are provided below.


W1/Q1: The main comparison lacks closely related task-aware block-pruning baselines, especially TaBP and TELL-TALE.

We have added TaBP to the main accuracy and efficiency comparisons. ReTaPER exceeds TaBP by 4.63 and 8.93 percentage points in nine-task average accuracy on Llama-3.1-8B and Qwen3-8B, respectively, without post-pruning fine-tuning.

TaBP derives task-specific pruning plans from output-distribution shifts. Its calibration uses up to 2,500 labeled examples per downstream task, with the data settings now specified in Appendix D.2. The following pruning-only comparisons use the main experiment's 25% target parameter-pruning ratio. Accuracy is reported in percent; Average is the unweighted mean over the nine tasks.

Llama-3.1-8B, without fine-tuning.

TaskTaBPIG-PruningReTaPER
PubMedQA55.2066.8074.00
MedMCQA24.3429.9823.86
LICQ46.1743.5857.75
FIQA-SA73.9346.5889.74
PIQA67.0873.2264.47
HellaSwag51.1065.3953.86
WinoGrande59.1961.0962.27
ARC-E47.5660.2044.07
ARC-C39.0834.4735.32
Average51.5253.4856.15

Qwen3-8B, without fine-tuning.

TaskTaBPIG-PruningReTaPER
PubMedQA58.6056.8064.20
MedMCQA22.5926.8030.61
LICQ46.1744.9254.17
FIQA-SA42.7445.7166.38
PIQA60.7772.2369.21
HellaSwag42.8457.7751.22
WinoGrande56.4358.6766.96
ARC-E46.4659.4947.23
ARC-C34.4734.3841.45
Average45.6750.7554.60

ReTaPER retains the highest average accuracy in these comparisons after adding TaBP. The task-level results also show that the advantage is not uniform: for example, IG-Pruning remains stronger on PIQA and ARC-E on both backbones. These results strengthen the comparison against task-aware pruning, while our claim remains an average-accuracy advantage over the evaluated baselines. Tables 1 and 3 and the corresponding experimental descriptions have been updated.

AuthorsOpenReview ↗

Response to Reviewer kLwB 2/6

W2/Q2 (task sensitivity): Is the average-accuracy advantage driven by FIQA-SA?

We added a complete leave-one-task-out analysis in Appendix E.2. For each omitted task, we recompute every method’s unweighted mean over the remaining eight tasks and reselect the strongest pruned baseline. The analysis uses Table 1 without retraining. Below, accuracy is in percent; Delta is ReTaPER minus the best baseline in percentage points.

Llama-3.1-8B

Omitted taskBest baseline (No FT)ReTaPER (No FT)DeltaBest baseline (FT)ReTaPER (FT)Delta
None (9 tasks)53.4856.15+2.6760.7066.97+6.27
PubMedQA51.8153.92+2.1059.6266.64+7.03
MedMCQA56.4260.19+3.7763.7668.46+4.70
LICQ54.7255.95+1.2362.3366.95+4.61
FIQA-SA54.3451.95-2.3957.3263.86+6.54
PIQA51.0155.11+4.1059.4766.20+6.73
HellaSwag51.9956.44+4.4561.0867.85+6.76
WinoGrande52.5355.38+2.8660.1866.70+6.52
ARC-E52.6457.66+5.0260.6066.03+5.43
ARC-C55.8658.75+2.9063.2470.06+6.82

Qwen3-8B

Omitted taskBest baseline (No FT)ReTaPER (No FT)DeltaBest baseline (FT)ReTaPER (FT)Delta
None (9 tasks)50.7554.60+3.8558.8864.53+5.65
PubMedQA50.0053.40+3.4159.3265.08+5.76
MedMCQA53.7557.60+3.8661.8865.94+4.06
LICQ51.4854.66+3.1858.8463.08+4.24
FIQA-SA51.3853.13+1.7554.7960.81+6.03
PIQA48.0752.78+4.7157.1163.80+6.68
HellaSwag49.8855.03+5.1559.6966.11+6.43
WinoGrande49.7653.06+3.3059.5664.19+4.64
ARC-E49.6655.53+5.8759.0464.43+5.39
ARC-C52.8056.25+3.4561.2367.37+6.14

Without fine-tuning, IG-Pruning is the strongest baseline in every subset. With fine-tuning, it is GradPruner on Llama except when omitting MedMCQA or LICQ (LLM-Streamline), and LLM-Streamline on Qwen except when omitting WinoGrande (GradPruner).

ReTaPER leads in 35 of 36 comparisons (97.22%; two backbones × two fine-tuning settings × nine omitted tasks). We acknowledge that the Llama pruning-only advantage is sensitive to FIQA-SA: removing it reverses the gap to −2.39 points. However, after recovery, every exclusion retains a positive gap, with minimum gains of 4.61 points on Llama and 4.06 on Qwen. These results support robustness across the tested task subsets while recognizing task-dependent performance. The overlapping subsets are sensitivity checks, not independent statistical trials.

AuthorsOpenReview ↗

Response to Reviewer kLwB 3/6

W2/Q2 (model generalization): Evaluation is limited to approximately 8B-parameter models, leaving generalization to larger or different architectures unclear.

We have extended evaluation to Qwen3-30B-A3B, which has a larger total parameter count and a mixture-of-experts architecture, complementing the two 8B dense backbones. The complete task-level results are shown below. Accuracy is in percent, and Average is the unweighted nine-task mean. Base Model and Tuned Base Model are unpruned references.

Qwen3-30B-A3B, without fine-tuning.

TaskBase ModelTaBPIG-PruningReTaPER
PubMedQA78.4074.0068.2075.80
MedMCQA52.1427.2530.7628.02
LICQ48.6344.2545.0858.25
FIQA-SA79.9157.6950.3188.32
PIQA80.4772.7473.6564.29
HellaSwag77.7552.3465.6354.26
WinoGrande70.7260.5461.9062.51
ARC-E79.0840.4960.1843.98
ARC-C55.8934.6435.0837.72
Average69.2251.5554.5357.02

Qwen3-30B-A3B, with fine-tuning.

TaskTuned Base ModelGradPrunerReTaPER
PubMedQA76.6058.4065.80
MedMCQA62.3146.5457.36
LICQ74.9161.4177.50
FIQA-SA88.3690.5393.97
PIQA84.7674.8972.83
HellaSwag78.0253.1255.79
WinoGrande71.7359.7669.22
ARC-E85.9463.3768.41
ARC-C63.2742.9344.78
Average76.2161.2267.30

Without fine-tuning, ReTaPER reaches 57.02 average accuracy, exceeding IG-Pruning by 2.49 points and TaBP by 5.47 points. After fine-tuning, it reaches 67.30, exceeding GradPruner by 6.08 points and leading on eight of nine tasks. Recovery raises ReTaPER's average from 57.02 to 67.30, including gains of 29.34 points on MedMCQA and 24.43 points on ARC-E.

These results extend the evidence to the tested larger MoE backbone. The gains remain task-dependent: IG-Pruning is stronger on PIQA, HellaSwag, and ARC-E without fine-tuning, and GradPruner remains stronger on PIQA after fine-tuning. The new results and analysis are included in Appendix F.1 (Table 12), with a corresponding main-text summary.

Reviewer_BWMqOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReTaPER, a task-aware structured-pruning method based on decomposing block importance into a generic component and a task-specific residual that approximately lies in a reusable low-rank subspace. The method learns this subspace and a set of anchor blocks from MMLU masking responses, reconstructs a new task’s full residual from 12 anchor evaluations, combines it with generic importance to determine a pruning order, and reuses the residual to allocate a fixed LoRA rank-unit budget. At 25% parameter pruning, it improves over the strongest reported baselines by 2.67–6.27 average accuracy points across Llama-3.1-8B and Qwen3-8B, with and without recovery.

Critical strengths

The central generic-plus-residual decomposition is intuitive and supported by extensive diagnostics. On Llama, the top eight residual directions explain 85.15% of the energy, compared with 45.32% and 75.28% under two randomized nulls; none of 2,000 null matrices reaches the observed top-eight concentration. The evaluation covers two backbones, nine tasks, five seeds, multiple pruning ratios, alternative reference-task collections, signal and anchor ablations, and explicit sparse-versus-complete probing. Particularly valuable is the transparent accuracy-efficiency result: using 13 rather than 65 evaluations per task reduces aggregate probing time from 219.97 to 43.90 minutes, a 5.01× speedup, while reporting the associated 3.74-point loss relative to complete marginal probing.

Critical weaknesses

Recovery is not parameter- or compute-matched. Dynamic allocations are constrained by summed rank units, although MHA and MLP projections have different dimensions, so equal rank sums need not imply equal trainable parameters or training FLOPs. To resolve this concern, the authors should rerun the uniform, generic-guided, combined-guided, residual-guided, and recovery-baseline comparisons with identical trainable-parameter counts, adapter target modules, training examples, optimizer budgets, and hyperparameter-search budgets. Please report five-seed task-level and macro accuracies with paired confidence intervals, actual trainable parameters, training FLOPs or wall time, and peak memory. This concern would be substantially resolved if residual guidance retains a statistically positive advantage under these controls.

The residual prior appears sensitive to reference-task composition and is validated only on a narrow task family. End-to-end results use two 8B backbones and multiple-choice tasks, while Table 4 shows that changing the reference collection can alter individual results sharply: the joint basis loses 12.20 points on PubMedQA and 15.67 on LICQ relative to the MMLU basis despite similar average accuracy. A resolving experiment should compare at least three independently constructed reference banks, repeated random half-subsets of each bank, and leave-one-domain-out construction, while fixing k=8, q=12, 200 probes, and the 25% pruning ratio. Report mean and worst-case downstream accuracy, variance across reference subsets, residual Pearson correlation, pruning-order Spearman correlation, selected-block overlap, and gap to full scanning. Including at least one generative or otherwise non-multiple-choice target family would determine whether the claimed reusable structure transfers beyond the current benchmark format.

Questions for authors
  1. Please provide the exactly parameter- and compute-matched recovery experiment described above. Does residual-guided allocation still beat uniform allocation and the strongest recovery baseline with a paired 95% confidence interval entirely above zero?

  2. Please provide the controlled reference-bank resampling and leave-one-domain-out experiment described above, including a non-multiple-choice target family. Does ReTaPER remain ahead of the strongest pruning baseline across reference subsets without large worst-case task regressions?

3 comments
AuthorsOpenReview ↗

Response to Reviewer BWMq 1/2

We thank the reviewer for the helpful comments. We have updated the manuscript in response, with revisions highlighted in blue, and provide detailed point-by-point responses below.


W1/Q1: Recovery allocation strategies and recovery baselines should be compared under matched parameter and compute budgets, with identical target modules, five-seed results, paired confidence intervals, and resource measurements.

We agree that equal rank sums do not imply equal parameter counts. The revised controls match each task to its native Residual parameter budget using the sum of r × (input dimension + output dimension) over target projections, averaging approximately 7.115M parameters per task. One rank across an MHA block costs 26,624 parameters; across an MLP block it costs 55,296. Under these parameter-matched settings, we reran the allocation ablation in Table 2(b).

Parameter-matched internal controls.

StrategyComplete allocation acc. (%)Fixed-target acc. (%)Complete-allocation train time (min)
Uniform65.25 ± 0.1365.11 ± 0.0815.97
Generic64.94 ± 0.1265.18 ± 0.1415.01
Combined65.55 ± 0.0967.05 ± 0.1515.01
Residual66.97 ± 0.1066.97 ± 0.1015.62

Accuracies are mean ± SD over five seeds; time sums nine training runs. Complete allocation allows all 43 retained blocks as candidates; adapted locations can differ. The fixed-target control uses exactly Residual's enabled projections, with positive rank at every target, to isolate rank distribution within that set.

Residual minus comparator (pp)Complete allocation: mean [95% CI]Fixed targets: mean [95% CI]
Uniform+1.72 [1.45, 1.98]+1.86 [1.65, 2.06]
Generic+2.03 [1.86, 2.19]+1.79 [1.54, 2.03]
Combined+1.42 [1.22, 1.62]-0.08 [-0.35, 0.19]

Each interval uses five paired seed-wise nine-task macro differences and a two-sided t-interval with four degrees of freedom. Residual is above Uniform in both controls. With fixed targets, however, its interval against Combined crosses zero: these results support the complete allocation strategy, including module selection, rather than a uniformly better rank distribution.

Detailed settings, seed-wise macro results, and paired intervals are in Table 2(b) and Appendix G.1–G.3 (Tables 14–16).

Adaptive recovery baselines with fixed candidates. We added AdaLoRA and DoRA as recovery baselines. For each task, all methods recover the same ReTaPER-pruned model and can adapt only the projection modules selected by Residual. Residual allocates ranks using residual scores, Uniform distributes ranks as evenly as possible, and the adaptive baselines adjust their allocations during training. Within each comparison, the methods share the training configuration and target the same final adapter-parameter budget, averaging approximately 7.115M parameters per task.

Comparison with AdaLoRA

MethodAcc. (%)Train time (min)Peak memory (GiB)Maintained params. (M)
Residual68.1017.4422.247.115
Uniform66.1417.4222.247.115
AdaLoRA59.0827.3622.4111.657

Comparison with DoRA

MethodAcc. (%)Train time (min)Peak memory (GiB)Maintained params. (M)
Residual68.1317.0722.267.115
Uniform65.3917.1122.267.115
DoRA61.9320.1822.4513.919

Values are five-seed means. Accuracy is the nine-task macro average; time sums nine training runs. Peak memory is the training peak averaged over seeds, and maintained parameters describe training-stage capacity rather than final adapter capacity.

Residual exceeds AdaLoRA and DoRA by 9.02 and 6.20 percentage points, respectively, while using less training time and fewer maintained parameters. Its additional gains of 1.96 and 2.74 points over Uniform support residual-guided allocation for recovering these pruned models. Detailed settings and both candidate scopes are reported in Appendix G.4 (Table 17).

AuthorsOpenReview ↗

Response to Reviewer BWMq 2/2

W2/Q2: Does the residual prior transfer across reference subsets and excluded domains, including a generative task, without large task regressions?

We added controlled reference-subset and domain-exclusion experiments in Appendix B.3 (Table 5), and generative code-output prediction in Appendix F.2 (Table 13).

Reference-task subsampling. We retain k=8, q=12, 200 downstream calibration examples, and the 25% target parameter-pruning ratio. Downstream calibration data, pruning rules, and evaluation remain fixed. Each subset recomputes generic importance, normalization, count weights, the basis, and anchors from its retained columns. The joint bank is refitted from MMLU and BIG-bench; it is not an independent third data source. The reference-composition experiments use Llama-3.1-8B without recovery and report nine-task macro accuracy (%).

Reference collection (full → subset size)Full collectionFive subsets: mean ± SD
MMLU (57 → 28)56.1553.37 ± 1.49
BIG-bench (18 → 9)51.8552.87 ± 1.11
Joint (75 → 37)55.8353.25 ± 2.58

Joint subsets contain 28 MMLU subjects and 9 BIG-bench tasks. The opposite trends for BIG-bench and the other two collections show that adding reference tasks does not uniformly improve transfer.

Leave-one-domain-out construction. We remove predefined MMLU subject groups and refit the bank with the same procedure:

Excluded domainRetained subjectsNine-task average (%)Change (pp)
None5756.15--
Biomedical4753.37-2.78
Law5455.54-0.61
Finance/business5049.85-6.30
Computer science5355.99-0.16

Excluding law or computer science has a small effect on the macro average, whereas excluding finance/business reduces it by 6.30 points. FIQA-SA falls from 89.74 to 55.64 in that setting, a 34.10-point regression. Relative to the main IG-Pruning reference of 53.48, complete MMLU and Joint remain ahead, but the three half-subset means do not preserve that lead. We have accordingly clarified the dependence on reference composition and domain coverage in Appendix B.3 and the limitations in Appendix J.4.

Generative coding evaluation. We evaluate Llama-3.1-8B-Instruct on CRUXEval-O with the default MMLU reference collection and a 200/100/500 train/validation/test split. Outputs are generated greedily and scored using execution-based pass@1; detailed settings are in Appendix F.2.

MethodRecoveryPassed / 500Pass@1 (%)
Base Model--17835.6
IG-PruningNo20.4
ReTaPERNo61.2
GradPrunerYes16432.8
ReTaPERYes17234.4

ReTaPER reaches 34.4% after recovery, exceeding GradPruner by 1.6 percentage points. Recovery increases ReTaPER's pass@1 by 33.2 points, highlighting its importance for restoring code-output prediction capability after pruning. This adds evidence on a generative task under the default reference collection. Together, the experiments broaden the evaluated task format and quantify sensitivity to reference choice.

Reviewer_BWMqOpenReview ↗

Follow-up after author response

Thank you for the detailed response and additional controls. The parameter-matched experiments substantially address my recovery-budget concern. The reference-bank experiments also helpfully quantify the dependence on reference composition and domain coverage, which the revised limitations now acknowledge. The fixed-target results appropriately distinguish the benefit of the complete allocation strategy from that of rank distribution alone. I retain my original weak-acceptance rating.

Reviewer_q18UOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper finds that block importance across different tasks can be decomposed into shared importance and low-rank task-specific residuals. Based on this, ReTaPER learns a residual subspace offline, and only probes a few blocks for new tasks to estimate their importance. The estimated importance is then used for pruning and LoRA rank allocation. Experiments on Llama-3.1-8B and Qwen3-8B show better performance and lower adaptation cost compared to existing methods.

Critical strengths
  • The low-rank observation is interesting and makes sense. It provides a reasonable way to reduce the cost of task-aware pruning, without evaluating every block for each new task

  • Using the same task importance information for both pruning and LoRA recovery is a nice design. Experiments on nine tasks and two models show consistent improvements.

Critical weaknesses
  • The low-rank subspace is mainly learned from MMLU subjects. Not clear how well this generalizes to tasks that are very different from MMLU, especially coding or other specialized domains

  • The method combines task-aware pruning with dynamic LoRA rank allocation. Although there are ablations, its still not fully clear how much improvement comes from better pruning and how much from the recovery strategy.

Questions for authors
  1. What happens if a new task does not fit well into the learned residual subspace? Is there any way to detect this?

2, How much improvement remains if all pruning baselines use the same dynamic LoRA rank allocation strategy?

2 comments
AuthorsOpenReview ↗

Response to Reviewer q18U 1/2

We appreciate the reviewer's constructive feedback. We have revised the manuscript accordingly, with changes highlighted in blue. Our point-by-point responses are provided below.


W1/Q1: How well does the MMLU-derived residual subspace generalize to substantially different tasks, and can unreliable predictions on a new task be detected?

Generalization to generative coding. We added CRUXEval-O, which requires generating the output of a Python function rather than selecting an answer. Appendix F.2 reports execution-based Pass@1 on 500 test examples using Llama-3.1-8B-Instruct and the MMLU-derived basis.

SettingMethodPass@1 (%)
UnprunedBase Model35.6
Without recoveryIG-Pruning0.4
Without recoveryReTaPER1.2
With recoveryGradPruner32.8
With recoveryReTaPER34.4

ReTaPER exceeds the corresponding pruning baselines by 0.8 percentage points without recovery and 1.6 points after recovery. Both pruned models perform poorly before recovery, while ReTaPER recovers to 34.4%, compared with 35.6% for the unpruned model. These results extend the evaluation to generative coding while showing the importance of recovery in this setting.

Dependence on reference-task coverage. Appendix B.3 examines reference-subset resampling and domain removal. Removing biomedical, legal, finance/business, or computer-science subjects changes nine-task average accuracy from 56.15% to 53.37%, 55.54%, 49.85%, and 55.99%, respectively. Finance/business removal produces the largest decline, including a FIQA-SA decrease from 89.74% to 55.64%. Thus, reference coverage matters, and we do not assume equally reliable transfer to every new domain.

Detecting unreliable reconstruction on a new task. We added an optional reliability check in Appendix I.4. After fitting task coordinates from the usual 12 anchors, we measure four additional blocks excluded from fitting and compare their observed accuracy drops with the predictions.

We validate this signal across the nine downstream tasks and CRUXEval-O, using the remaining 48 blocks only to measure reconstruction error. Five task-independent selections of the four check blocks give the following correlations across ten tasks:

Check-set seedSpearman correlation95% bootstrap CI
420.794[0.185, 1.000]
1230.842[0.370, 1.000]
123450.467[−0.371, 0.963]
1234560.503[−0.324, 0.958]
1000.527[−0.317, 1.000]

The correlations are positive across all five selections, although three confidence intervals cross zero. These results support an exploratory reconstruction-reliability check with four additional block probes; they do not establish a universal alarm threshold or guarantee detection of final pruning-accuracy regressions.

AuthorsOpenReview ↗

Response to Reviewer q18U 2/2

W2/Q2: How much of the improvement comes from pruning versus recovery, and how much remains when all pruning baselines use the same dynamic LoRA strategy?

We separate the pruning-only comparison from controlled recovery experiments on the same pruned model.

Pruning without recovery. Table 1 reports the following nine-task average accuracies (%), so these differences arise before any LoRA recovery:

BackboneIG-PruningReTaPERDifference (pp)
Llama-3.1-8B53.4856.15+2.67
Qwen3-8B50.7554.60+3.85

Recovery with the pruning masks held fixed. We added five-seed controls on identical ReTaPER-pruned Llama-3.1-8B models. Each strategy targets Residual's per-task trainable-parameter budget, averaging approximately 7.115M parameters per task, with shared training data, update counts, optimizer settings, and LoRA scaling. Parameter counts account for the dimensions of the adapted projections, rather than summed ranks.

Allocation strategyAll retained blocks are candidatesFixed target modules
Uniform65.25 ± 0.1365.11 ± 0.08
Generic64.94 ± 0.1265.18 ± 0.14
Combined65.55 ± 0.0967.05 ± 0.15
Residual66.97 ± 0.1066.97 ± 0.10

Entries are mean ± SD of nine-task macro accuracy (%) over five seeds. In the first column, strategies can select different adapted blocks; in the second, every strategy adapts exactly Residual's enabled target projections with positive ranks.

For complete allocation, Residual improves over Uniform by 1.72 percentage points, with a paired 95% CI of [1.45, 1.98]. Its gains over Generic and Combined are 2.03 and 1.42 points, respectively. When target modules are fixed, Combined rises from 65.55 to 67.05, and the Residual-minus-Combined interval is [-0.35, 0.19]. This suggests an important role for selecting which blocks receive recovery capacity, without establishing that Residual's rank distribution is better than Combined's within that fixed set.

We have scoped the recovery conclusion to ReTaPER-pruned models and documented the controls in Table 2(b) and Appendix G.1–G.3 (Tables 14–16). Together, the pruning-only comparisons and controlled recovery experiments demonstrate an advantage before recovery and isolate the effects of recovery allocation on a shared pruned model.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Authors’ Responses and Discussion

Dear Area Chair and Reviewers,

We thank all three reviewers for their thoughtful feedback. During the rebuttal, we provided point-by-point responses and added experiments addressing baseline coverage, recovery controls, robustness, and generalization. The corresponding manuscript changes are highlighted in blue. We summarize the main updates below to facilitate the final assessment.

1. Stronger pruning comparisons. We added the closely related task-aware baseline TaBP to Table 1. ReTaPER retains the highest nine-task average accuracy among the compared pruned models on both dense backbones. Without recovery, it exceeds IG-Pruning by 2.67 and 3.85 percentage points on Llama-3.1-8B and Qwen3-8B, respectively, demonstrating an advantage before LoRA recovery.

2. Controlled recovery experiments. We repeated the allocation ablation with matched per-task trainable-parameter budgets, averaging approximately 7.115M parameters, and five paired seeds. Residual exceeds Uniform by 1.72 points, with a paired 95% CI of [1.45, 1.98]. Fixed-target controls show that the benefit includes selecting where to adapt; they do not establish that Residual's rank distribution outperforms Combined within the same target modules. We also added AdaLoRA and DoRA comparisons with their corresponding Residual and Uniform references (Table 2(b), Appendix G).

3. Task sensitivity and broader evaluation. Complete leave-one-task-out analysis preserves ReTaPER's lead over the strongest baseline in 35 of 36 comparisons (97.22%) across two backbones, two fine-tuning settings, and nine task exclusions (Appendix E.2). We added Qwen3-30B-A3B, extending evaluation to a larger MoE backbone, and generative code-output prediction on CRUXEval-O. On the MoE model, ReTaPER exceeds IG-Pruning by 2.49 points without recovery and GradPruner by 6.08 points after recovery; on CRUXEval-O, recovered ReTaPER reaches 34.4% Pass@1 versus GradPruner's 32.8% (Appendix F).

4. Calibration data, transfer, and practical scope. We documented calibration/recovery data and gave pruning baselines access to the same MMLU pool; the strongest such baseline reaches 54.28%, below ReTaPER's 56.15% pruning-only accuracy (Appendix D.2). Reference-subset resampling and domain exclusion quantify sensitivity to reference composition, and an optional four-block check provides an exploratory indicator of new-task reconstruction reliability (Appendices B.3 and I.4). We explicitly acknowledge domain-coverage limitations and the substantial upfront calibration cost, clarifying the accuracy-efficiency trade-off (Appendix J.4).

In the subsequent discussion, Reviewer BWMq stated that the parameter-matched experiments substantially address the recovery-budget concern, welcomed the reference-composition analysis, and endorsed the distinction between complete allocation and rank distribution alone. The reviewer retained their weak-acceptance rating. At the time of this summary, we have not received further feedback from Reviewers q18U or kLwB.

We appreciate the Area Chair's and reviewers' time and consideration of the additional evidence and clarified scope.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.