acceptodds
Under review as a conference paper at ICLR 2027

HyperRepair: Recovering Forgotten General Capabilities of Fine-Tuned Language Models

Abstract

Fine-tuning improves a language model's performance in a target domain but can weaken its general capabilities. Existing methods often face a trade-off between general capabilities and domain specialization, making it difficult to preserve both. We introduce HyperRepair, a framework for recovering general capabilities while retaining domain specialization at only inference cost. To make recovery generalizable across models sharing a pretrained backbone, HyperRepair learns a hypernetwork from repairs trained on source models. The hypernetwork uses activation fingerprints that summarize changes from the pretrained backbone to generate a tailored Repairing Side Net (RSN), whose prompt-conditioned gate applies corrections selectively. This enables recovery on models from unseen adaptation domains without target-domain examples or additional optimization. We evaluate HyperRpair across two model families, 12 LoRA adaptation datasets, and 15 general benchmarks. On the default split, HyperRepair achieves 93.4% of the base models' average general score while retaining 99.5% of the fine-tuned models' average domain score.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
large language models, catastrophic forgetting, continual learning, fine-tuning, hypernetworks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Pmn2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HyperRepair for restoring general-task performance of fine-tuned language models while retaining domain specialization. It trains gated Repairing Side Nets (RSNs) for source models, then learns a hypernetwork that generates repair parameters from activation fingerprints. For a model adapted to a held-out domain, repair generation requires neither target-domain examples nor additional optimization. Experiments evaluate two backbones across twelve adaptation domains and fifteen general benchmarks, with ablations examining fingerprint conditioning, functional supervision, and selective gating.

Critical strengths
  1. The paper addresses an important problem: domain specialization can degrade general capabilities, while the original adaptation data may no longer be available. Generating a repair without target-domain optimization is a useful objective. The frozen-backbone design and prompt-conditioned correction provide a coherent approach to balancing general recovery with domain retention.
  2. HyperRepair improves the average general score of all six held-out models in Table 1, while Figure 3 shows that domain performance is largely retained. The shared-repair and always-on-gate ablations provide useful evidence for model-specific conditioning and selective correction. These results make the approach promising despite the uneven recovery across individual benchmarks.
Critical weaknesses
  1. Table 1 supports partial, task-dependent recovery rather than consistently restored capabilities. For Qwen adapted to SNIPS, MBPP reaches only 23.2 versus the base model’s 63.8; for Qwen adapted to DROP, repair reduces MBPP from 58.8 to 56.8. The headline average can therefore mask substantial residual deficits and regressions. Category-level summaries, base-relative gaps, and multiple-seed results would clarify robustness.
  2. As a minor presentation issue, Table 2’s dense wrapping and equation placement make the three-way comparison harder to scan; clearer column spacing and response separation would help.
Questions for authors

Appendix Table 5 shows that repair training uses labeled data from the evaluated general-task families, including proxy data for code generation. This warrants distinguishing supervised general-task learning from recovery that transfers beyond those families. Figure 4 already includes a shared-RSN control, but its aggregate result on one backbone and split offers limited insight into where conditioning helps.

4 comments

We thank the reviewer Pmn2 for the valuable feedback, which has helped us identify opportunities to improve the clarity and rigor of our work.

W1 To check whether averages hide inconsistent recovery, we added task-level, multi-seed results on Qwen2.5-1.5B with rank-64/two-epoch LoRA and full fine-tuning, reporting fine-tuned, repaired, and Base scores. In the LoRA setting, MBPP improves from 28.53 to 38.60 ± 0.64, remaining 5.60 points below Base at 44.20, while the general average rises from 47.15 to 54.50 ± 0.11. We will summarize these results by capability and discuss your original SNIPS→MBPP deficit of 40.6 points and DROP→MBPP regression of 2.0 points separately, keeping the evidence from the two backbones distinct. Thus, the results support aggregate recovery in the tested settings while identifying residual task-level deficits and individual regressions. The additional Qwen2.5-1.5B results are included in Appendix H.2 of the revised paper.

BenchmarkBaseLoRA rank16/epoch2LoRA rank16/epoch2 + HyperRepairLoRA rank64/epoch2LoRA rank64/epoch2 + HyperRepairLoRA rank64/epoch4LoRA rank64/epoch4 + HyperRepairFull Fine-tuning epoch2Full fine-tuning epoch2 + HyperRepair
ARC-Challenge44.9748.5554.5542.6351.71 ± 0.3242.3249.5242.9251.36 ± 0.22
HellaSwag67.8467.0976.1262.8373.56 ± 0.3461.4171.9663.9174.86 ± 0.36
MMLU59.7059.0658.3034.9452.53 ± 0.5828.5041.2845.8751.75 ± 0.54
OpenBookQA40.2040.6045.8040.2746.49 ± 0.1940.0046.2741.6046.87 ± 0.71
PIQA75.9575.9777.9373.6377.14 ± 0.3573.1475.9374.0877.48 ± 0.21
SciQ93.3095.4396.3088.4794.12 ± 0.4087.8093.1391.8394.53 ± 0.19
WinoGrande63.6963.7768.1464.9368.34 ± 0.1362.9065.8064.6168.91 ± 0.14
ANLI-R139.5035.0045.4332.0042.69 ± 0.2530.9040.1731.4041.22 ± 0.31
COPA83.0085.6785.6782.6785.33 ± 1.4583.6784.0082.6785.11 ± 1.50
Social IQa49.1350.0553.2847.4450.44 ± 0.0646.6749.1847.8350.34 ± 0.33
LogiQA31.0332.1032.7230.4731.23 ± 0.1828.7829.8530.2630.67 ± 0.59
WiC52.9854.1351.9349.6351.86 ± 0.9849.9551.2049.5851.41 ± 0.24
MATH28.3627.2728.301.8320.38 ± 0.270.0320.6315.1124.96 ± 0.24
HumanEval37.8037.2037.5227.0333.08 ± 1.0020.9330.9332.3233.20 ± 1.00
MBPP44.2042.1342.2028.5338.60 ± 0.6419.8728.0736.5341.80 ± 0.33
Average54.1154.2756.9547.1554.50 ± 0.1145.1251.8650.0354.96 ± 0.10

W2

Following your suggestion, we have revised Table 2 in the updated PDF by increasing column spacing, improving text wrapping, and more clearly separating the outputs of the base model, the fine-tuned model, and HyperRepair. We have also adjusted the placement of the formulas in the MATH example, making the three-way comparison easier to scan.

Q1

To examine recovery beyond the benchmarks used for general-task supervision, we added a Qwen2.5-1.5B experiment training RSNs and HyperNet on six general benchmarks and completely excluding nine others from both training stages. On these nine unseen benchmarks, the average improves from 55.69 to 59.66 ± 0.14. We distinguish this benchmark-level holdout from task-family novelty: HumanEval and MBPP, for example, both evaluate code generation. Thus, these results demonstrate generalization to benchmarks unseen by both repair-training stages, with task-family overlap explicitly distinguished. We have included these results in Appendix H.1 of the revised paper.

BackboneTrained or held outEvaluated general benchmarkBaseFine-tunedHyperRepair(mean ± std)
Qwen2.5-1.5Bheld outARC-Challenge44.9742.6350.31 ± 0.10
Qwen2.5-1.5BtrainedHellaSwag67.8462.8376.18 ± 0.15
Qwen2.5-1.5BtrainedMMLU59.7034.9452.42 ± 0.75
Qwen2.5-1.5Bheld outOpenBookQA40.2040.2745.82 ± 0.80
Qwen2.5-1.5Bheld outPIQA75.9573.6377.03 ± 0.09
Qwen2.5-1.5Bheld outSciQ93.3088.4793.31 ± 0.20
Qwen2.5-1.5Bheld outWinoGrande63.6964.9367.20 ± 0.17
Qwen2.5-1.5BtrainedANLI-R139.5032.0042.49 ± 1.31
Qwen2.5-1.5Bheld outCOPA83.0082.6784.89 ± 0.19
Qwen2.5-1.5BtrainedSocial IQA49.1347.4449.95 ± 0.16
Qwen2.5-1.5Bheld outLogiQA31.0330.4731.78 ± 0.06
Qwen2.5-1.5Bheld outWiC52.9849.6351.76 ± 0.49
Qwen2.5-1.5BtrainedMATH28.361.8319.07 ± 0.10
Qwen2.5-1.5BtrainedHumanEval37.8027.0337.63 ± 0.41
Qwen2.5-1.5Bheld outMBPP44.2028.5334.84 ± 0.63
Qwen2.5-1.5Btrainedtrained 6 general benchmark average score47.0534.3546.29 ± 0.37
Qwen2.5-1.5Bheld outheld out 9 general benchmark average score58.8155.6959.66 ± 0.14
Qwen2.5-1.5Ball15 general benchmark average score54.1147.1554.31 ± 0.22

Q2

To show where conditioning helps beyond the aggregate ablation, we added shared-versus-conditioned RSN comparisons on split D for SNIPS-adapted Llama3 and LEDGAR-adapted Qwen2.5-1.5B. The shared control retains HyperNet training and RSN generation, so this comparison isolates conditioning on the target model. Conditioning raises MBPP from 0 to 16.8 for Llama3 and MMLU from 35.12 to 53.11 for Qwen2.5-1.5B, showing which adaptation-task pairs benefit from conditioning beyond the shared repair. Thus, conditioning improves repair for specific adaptation-task pairs across the two evaluated backbones, beyond the shared-RSN control.

BackboneSplitdomain taskgeneral taskfine-tunedw/o fingerprintHyperRepairConditioning benifits
Llama3.1-8BDSNIPS-7MBPP1.2016.816.8
Llama3.1-8BDSNIPS-7MATH009.49.4
Qwen2.5-1.5BDLEDGARMMLU24.5935.1253.1117.99
Qwen2.5-7BAHateXplainCOPA7585927
Qwen2.5-7BCRACEOpenBookQA4850.257.47.2

Thank you for the additional experiments and the revisions to Table 2. The task-level, multi-seed results clarify where recovery is consistent and where substantial gaps remain. The gains on benchmarks excluded from both repair-training stages strengthen the case for transfer to unseen benchmarks, and the shared-RSN comparison shows that fingerprint conditioning can help on specific task pairs. This evidence addresses my main concerns sufficiently for me to raise my rating from 2 to 3.

Dear Reviewer Pmn2,

Thank you for taking the time to review our additional experiments and revised manuscript, and for your positive reassessment. Your suggestions helped us present a more precise account of both the benefits and limitations of HyperRepair.

We appreciate your thoughtful and constructive engagement throughout the discussion.

Best,

Authors

Reviewer_9Ly8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper uses the difference in activation values across layers between the pretrained model and the fine-tuned model to generate a gated side network to recover general capabilities while preserving the domain specialization. The experiments on 12 LoRA adaptation datasets and 15 general benchmarks show that the proposed method achieves 93% of the average general score while retaining 99.5% of the fine-tuned domain score.

Critical strengths
  1. The paper introduces a useful formulation of amortised post-hoc repair across adapted models.
  2. The experiments for evaluation are broad, encompassing 2 backbone families, 12 LoRA datasets and 15 general benchmarks. The results demonstrate substantial recovery of general capabilities while largely preserving domain performance.
  3. The ablation studies are comprehensive and support the necessity of the gate and loss function.
Critical weaknesses
  1. The biggest concern is that the training of RSN and HyperNet requires labelled training data from general datasets. Therefore, the source of improvement is not clear. The experiments for now are based on RSN and the training data. An ablation study training the fine-tuned model on the same general data without RSN is required and helpful.
  2. The comparison miss the relevant methods like weight interpolation, LiNeS, rouing method. More comparisons are required to demonstrate superiority rather than comparison over simpler methods.
  3. The experiments use a fixed LoRA configuration and seed, providing limited evidence of robustness to different adaptation settings, such as full parameter fine-tuning, even on a small language model.
  4. The claims regarding efficiency require further quantifiable results; specifically, the following data should be provided: decoding overhead, as well as the training costs for RSN and hypernetworks.
Questions for authors

refer to the weaknesses.

4 comments

We sincerely thank the reviewer 9Ly8 for the careful reading and thoughtful comments. Below, we provide clarifications and additional evidence to address their concerns.

W1

To test whether the gains extend beyond the supervised benchmarks, we added a Qwen2.5-1.5B experiment that trains both RSNs and HyperNet on six general benchmarks and reserves nine exclusively for evaluation. These nine benchmarks are completely excluded from both training stages, and their average improves from 55.69 to 59.66 ± 0.14. General supervision learns a reusable HyperNet offline, so applying it to a new target requires no further optimization. Thus, the improvement transfers to unseen general benchmarks, complementing the direct continued-training control below. We have included these results in Appendix H.1 of the revised paper.

BackboneTrained or held outEvaluated general benchmarkBaseFine-tunedHyperRepair(mean ± std)
Qwen2.5-1.5Bheld outARC-Challenge44.9742.6350.31 ± 0.10
Qwen2.5-1.5BtrainedHellaSwag67.8462.8376.18 ± 0.15
Qwen2.5-1.5BtrainedMMLU59.7034.9452.42 ± 0.75
Qwen2.5-1.5Bheld outOpenBookQA40.2040.2745.82 ± 0.80
Qwen2.5-1.5Bheld outPIQA75.9573.6377.03 ± 0.09
Qwen2.5-1.5Bheld outSciQ93.3088.4793.31 ± 0.20
Qwen2.5-1.5Bheld outWinoGrande63.6964.9367.20 ± 0.17
Qwen2.5-1.5BtrainedANLI-R139.5032.0042.49 ± 1.31
Qwen2.5-1.5Bheld outCOPA83.0082.6784.89 ± 0.19
Qwen2.5-1.5BtrainedSocial IQa49.1347.4449.95 ± 0.16
Qwen2.5-1.5Bheld outLogiQA31.0330.4731.78 ± 0.06
Qwen2.5-1.5Bheld outWiC52.9849.6351.76 ± 0.49
Qwen2.5-1.5BtrainedMATH28.361.8319.07 ± 0.10
Qwen2.5-1.5BtrainedHumanEval37.8027.0337.63 ± 0.41
Qwen2.5-1.5Bheld outMBPP44.2028.5334.84 ± 0.63
Qwen2.5-1.5Btrainedtrained 6 general benchmark average score47.0534.3546.29 ± 0.37
Qwen2.5-1.5Bheld outheld out 9 general benchmark average score58.8155.6959.66 ± 0.14
Qwen2.5-1.5Ball15 general benchmark average score54.1147.1554.31 ± 0.22

Following your suggested control, we continued training the Qwen2.5-1.5B fine-tuned model on the same general data for the same number of steps on LEDGAR domain, without an RSN. Direct training achieves a higher general score than HyperRepair (64.95 versus 53.38), while HyperRepair retains more domain performance (98.77 versus 94.88). The comparison also captures the cost of applying each method to a target: direct training requires 2.503 GPU-hours of optimization, whereas the reusable HyperNet requires 5.51 seconds of preparation. Thus, HyperRepair offers stronger domain retention and avoids target-specific optimization in this comparison. We have included these results in Appendix H.3 of the revised paper.

ModelMethodGeneral scoreDomain scoreAdditional target training (GPU-hours)Target preparation time (s)
qwen2.5-1.5bFine-tune45.2899.0000
qwen2.5-1.5bFine-tune + general data training64.9594.882.5039012.37 (2.503 hours)
qwen2.5-1.5bHyperRepair53.3898.7705.51

W2

We added your suggested weight interpolation, LiNeS, and routing baselines on Qwen2.5-1.5B, using the same target models and evaluation setting to compare recovery, domain retention, and application costs. HyperRepair reaches 54.50 ± 0.11 on General15, compared with 53.90, 50.69, and 53.95 for these baselines, respectively, while its domain score is 85.51 versus 85.03 before repair. The comparison shows different trade-offs: interpolation and LiNeS have lower preparation costs and higher domain scores, while HyperRepair achieves the highest general average. Thus, HyperRepair achieves the highest general average among the tested baselines while maintaining average domain performance. We have included these results in Appendix H.4 of the revised paper.

BackboneMethodGeneral15 score (%)Domain score (%)Additional preparation time (s)Decoding latency (ms/token)
Qwen2.5-1.5BBase54.1147.07052.69
Qwen2.5-1.5Bfine-tuning47.1585.030109.77
Qwen2.5-1.5BWeight interpolation53.9085.780.000036110.67
Qwen2.5-1.5BLiNeS50.6986.220.000033110.70
Qwen2.5-1.5BRouting53.9585.50601.31120.56
Qwen2.5-1.5BHyperRepair54.50 ± 0.1185.51 ± 0.005.51116.38

W3

Following your suggestion to test beyond a fixed LoRA recipe, we added Qwen2.5-1.5B experiments with rank 16/two epochs, rank 64/two epochs, rank 64/four epochs, and two-epoch full fine-tuning. We applied the already-trained HyperNet directly to these models without retraining it for each configuration. General performance improves in all four settings, including 45.12 to 51.86 after four-epoch LoRA training and 50.03 to 54.96 ± 0.10 after full fine-tuning, supporting applicability across these adaptation settings. The results show that the repair learned by HyperNet applies beyond a single LoRA recipe, including the tested full-parameter fine-tuning setting. We have included these results in Appendix H.2 of the revised paper.

BenchmarkBaseLoRA rank16/epoch2LoRA rank16/epoch2 + HyperRepairLoRA rank64/epoch2LoRA rank64/epoch2 + HyperRepairLoRA rank64/epoch4LoRA rank64/epoch4 + HyperRepairFull Fine-tuning epoch2Full fine-tuning epoch2 + HyperRepair
ARC-Challenge44.9748.5554.5542.6351.71 ± 0.3242.3249.5242.9251.36 ± 0.22
HellaSwag67.8467.0976.1262.8373.56 ± 0.3461.4171.9663.9174.86 ± 0.36
MMLU59.7059.0658.3034.9452.53 ± 0.5828.5041.2845.8751.75 ± 0.54
OpenBookQA40.2040.6045.8040.2746.49 ± 0.1940.0046.2741.6046.87 ± 0.71
PIQA75.9575.9777.9373.6377.14 ± 0.3573.1475.9374.0877.48 ± 0.21
SciQ93.3095.4396.3088.4794.12 ± 0.4087.8093.1391.8394.53 ± 0.19
WinoGrande63.6963.7768.1464.9368.34 ± 0.1362.9065.8064.6168.91 ± 0.14
ANLI-R139.5035.0045.4332.0042.69 ± 0.2530.9040.1731.4041.22 ± 0.31
COPA83.0085.6785.6782.6785.33 ± 1.4583.6784.0082.6785.11 ± 1.50
Social IQa49.1350.0553.2847.4450.44 ± 0.0646.6749.1847.8350.34 ± 0.33
LogiQA31.0332.1032.7230.4731.23 ± 0.1828.7829.8530.2630.67 ± 0.59
WiC52.9854.1351.9349.6351.86 ± 0.9849.9551.2049.5851.41 ± 0.24
MATH28.3627.2728.301.8320.38 ± 0.270.0320.6315.1124.96 ± 0.24
HumanEval37.8037.2037.5227.0333.08 ± 1.0020.9330.9332.3233.20 ± 1.00
MBPP44.2042.1342.2028.5338.60 ± 0.6419.8728.0736.5341.80 ± 0.33
Average54.1154.2756.9547.1554.50 ± 0.1145.1251.8650.0354.96 ± 0.10

W4

For your efficiency question, we added Qwen2.5-1.5B measurements of offline HyperNet training and new-target preparation, plus decoding comparisons of LoRA and fully fine-tuned models with and without RSNs. In the General15 setting, HyperNet averages 3.17 GPU-hours per 3,000-step seed on one RTX PRO 6000 Blackwell, including preparation and saving; new-target repair preparation takes 5.51 seconds without further optimization. The paired decoding comparisons show latency increasing from 109.77 to 116.38 ms/token for LoRA (+6.02%) and from 52.70 to 56.97 ms/token for full fine-tuning (+8.10%). Thus, after offline training, HyperRepair requires no target-specific optimization and adds 6.02–8.10% decoding latency in the measured settings. The target-preparation and LoRA decoding results are included in Appendix H.4 of the revised paper.

Cost itemHyperNet
Training setting9-source / General15
GPU per training job1 × RTX PRO 6000 Blackwell
Trainable parameters21.28M
Training steps per seed3,000
Mean training time per seed3.17 hours
Mean training cost per seedapproximately 3.17 GPU-hours
Average per-run peak PyTorch allocated memory39.42 GiB
Training time for the three seeds3.28, 3.60, 2.65 hours
Cumulative recorded cost across three seeds9.52 GPU-hours
Timing scopeIncludes preparation and checkpoint saving
Concurrent executionSome runs overlap; cumulative job GPU-hours are not total elapsed wall-clock time
MethodTTFT (ms)Decoding latency (ms/token)Throughput (tokens/s)GPU utilization (%)Sampled peak GPU memory (GiB)
LoRA fine-tuning4662.29109.772824.0289.4630.03
LoRA fine-tuning + RSN4722.25116.382711.1189.5930.05
Full fine-tuning2041.1452.706073.8085.7729.42
Full fine-tuning + RSN2092.4056.975739.7284.2529.60

Dear Reviewer 9Ly8,

Thank you again for your helpful suggestions. As the discussion period is drawing to a close, we would like to briefly follow up on our rebuttal. Following your recommendations, we added a same-data continued-training control without an RSN, evaluations on held-out general benchmarks, and comparisons with weight interpolation, LiNeS, and routing. We also tested different LoRA configurations and full fine-tuning, and measured HyperNet training, target preparation, and decoding costs.

We would appreciate your acknowledgment of our responses and revisions, as well as any further feedback. If these additions and revisions have addressed your main concerns, we would be grateful if you could consider raising your score. Please let us know if any concerns remain or any points require further clarification so we can address them promptly. Thank you for your time and consideration.

Best,

Authors

Dear Reviewer 9Ly8,

As the rebuttal period is nearing its end, we would greatly appreciate any feedback you could provide on our responses and additional experiments, so that we have time to address any remaining concerns.

Best,

Authors

Reviewer_HBzJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes HyperRepair, a framework for recovering general capabilities degraded by fine-tuning while preserving domain-specific performance. The method first trains Repairing Side Nets (RSNs) for source models and then learns a hypernetwork that generates model-specific RSN parameters from activation fingerprints. A prompt-conditioned gate selectively applies the corrections during inference. Experiments on two LLM backbones, 12 adaptation datasets, and 15 general benchmarks show that HyperRepair improves general performance while largely retaining domain-specific capabilities on held-out adaptation domains.

Critical strengths
  1. The paper addresses the practical and important problem of recovering general capabilities in already fine-tuned LLMs without additional optimization on each target model.

2.Well-motivated framework. The combination of activation fingerprints, hypernetwork-based parameter generation, and prompt-conditioned gating provides an interesting approach to transferring repair knowledge across fine-tuned models.

  1. Solid experimental evaluation. The authors evaluate two model families across 12 adaptation domains, using four held-out-domain splits. The results demonstrate improvements over the selected baselines, and the ablation studies provide useful evidence for the contributions of fingerprint conditioning, functional supervision, and gating.
Critical weaknesses
  1. Limited evidence for the generalization of HyperNet across fine-tuned models. Each HyperNet is trained using only nine source models, despite generating RSNs containing approximately 10 million parameters. Although low-rank parameter generation and functional supervision help constrain this problem, it remains unclear how reliably the learned fingerprint-to-repair mapping generalizes beyond these limited source models. Experiments varying the number of source models and fine-tuning configurations would strengthen this claim.

  2. The necessity of model-specific parameter generation seems limited. The ablation in Figure 4 shows that removing fingerprint conditioning reduces the average general score by only 2.24 points on Qwen2.5-7B. This suggests that a shared RSN already provides much of the improvement. More extensive comparisons would help establish whether the additional complexity of HyperNet is justified by its model-specific adaptation.

  3. Missing comparisons with simpler inference-time alternatives. Since the experiments use LoRA-adapted models with an available pretrained backbone, selectively disabling the LoRA adapter or routing between the pretrained and fine-tuned models represents a natural alternative. A learned prompt router or conditional interpolation baseline could help determine whether generating a separate RSN is necessary to achieve the reported recovery–retention trade-off.

  4. Generalization to unseen general tasks is not discussed and demonstrated enough. The source RSNs and HyperNet are trained using labeled examples from the same 15 general benchmarks used for evaluation, although the test examples are held out. Therefore, the results primarily demonstrate transfer to unseen adaptation domains rather than recovery of general capabilities on unseen tasks. Evaluation on additional general benchmarks excluded from repair training would provide stronger evidence.

Questions for authors

1.Can HyperRepair recover performance on general benchmarks that are completely excluded from both RSN and HyperNet training?

2.How well does the learned HyperNet generalize across different LoRA ranks, fine-tuning strengths, and independently trained checkpoints from the same adaptation domain?

3.What are the end-to-end training and inference costs compared with simpler post-hoc recovery approaches?

7 comments

We thank the reviewer HBzJ for the time and constructive feedback. We address the main concerns below.

W1

With nine source models, we learn a constrained mapping from model fingerprints to low-rank corrections of a shared RSN template. The fingerprints describe fine-tuning-induced activation changes, while weight reconstruction, representation alignment, and functional supervision constrain the correction each model needs. Our original experiments evaluate this mapping on 24 unseen fine-tuned models across four held-out splits and two backbones.

Following your suggestion to vary the adaptation settings, we added Qwen2.5-1.5B experiments with LoRA ranks 16 and 64, two- and four-epoch fine-tuning, and full fine-tuning, applying the existing HyperNet to these target models. These experiments test transfer beyond one fixed LoRA configuration. HyperTuning[1] and HINT[2] provide related examples of transferable conditional parameter generation. Thus, the evidence supports transfer of the learned fingerprint-to-repair mapping to unseen fine-tuned models and to the additional adaptation settings tested. These results are included in Appendix H.2 of the revised paper.

[1] Phang, Jason, et al. "Hypertuning: Toward adapting large language models without back-propagation." International Conference on Machine Learning. pmlr, 2023.

[2] Ivison, Hamish, et al. "HINT: hypernetwork instruction tuning for efficient zero-and few-shot generalisation." Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2023.

BenchmarkBaseLoRA rank16/epoch2LoRA rank16/epoch2 + HyperRepairLoRA rank64/epoch2LoRA rank64/epoch2 + HyperRepairLoRA rank64/epoch4LoRA rank64/epoch4 + HyperRepairFull Fine-tuning epoch2Full fine-tuning epoch2 + HyperRepair
ARC-Challenge44.9748.5554.5542.6351.71 ± 0.3242.3249.5242.9251.36 ± 0.22
HellaSwag67.8467.0976.1262.8373.56 ± 0.3461.4171.9663.9174.86 ± 0.36
MMLU59.7059.0658.3034.9452.53 ± 0.5828.5041.2845.8751.75 ± 0.54
OpenBookQA40.2040.6045.8040.2746.49 ± 0.1940.0046.2741.6046.87 ± 0.71
PIQA75.9575.9777.9373.6377.14 ± 0.3573.1475.9374.0877.48 ± 0.21
SciQ93.3095.4396.3088.4794.12 ± 0.4087.8093.1391.8394.53 ± 0.19
WinoGrande63.6963.7768.1464.9368.34 ± 0.1362.9065.8064.6168.91 ± 0.14
ANLI-R139.5035.0045.4332.0042.69 ± 0.2530.9040.1731.4041.22 ± 0.31
COPA83.0085.6785.6782.6785.33 ± 1.4583.6784.0082.6785.11 ± 1.50
Social IQa49.1350.0553.2847.4450.44 ± 0.0646.6749.1847.8350.34 ± 0.33
LogiQA31.0332.1032.7230.4731.23 ± 0.1828.7829.8530.2630.67 ± 0.59
WiC52.9854.1351.9349.6351.86 ± 0.9849.9551.2049.5851.41 ± 0.24
MATH28.3627.2728.301.8320.38 ± 0.270.0320.6315.1124.96 ± 0.24
HumanEval37.8037.2037.5227.0333.08 ± 1.0020.9330.9332.3233.20 ± 1.00
MBPP44.2042.1342.2028.5338.60 ± 0.6419.8728.0736.5341.80 ± 0.33
Average54.1154.2756.9547.1554.50 ± 0.1145.1251.8650.0354.96 ± 0.10

W2

To examine where model-specific conditioning adds value, we added comparisons between shared and conditioned RSNs on split D for LEDGAR-adapted Qwen2.5-1.5B and SNIPS-adapted Llama3. Both variants retain HyperNet training and RSN generation with the same losses and budget, isolating the conditioning effect. Conditioning raises MMLU from 35.12 to 53.11 in the former setting and MBPP from 0 to 16.8 in the latter, showing where the shared repair leaves a gap that model-specific correction addresses. Thus, model-specific conditioning provides additional recovery over a shared RSN for the tested model-task pairs.

BackboneSplitdomain taskgeneral taskfine-tunedw/o fingerprintHyperRepairConditioning benifits
Llama3.1-8BDSNIPS-7MBPP1.2016.816.8
Llama3.1-8BDSNIPS-7MATH009.49.4
Qwen2.5-1.5BDLEDGARMMLU24.5935.1253.1117.99
Qwen2.5-7BAHateXplainCOPA7585927
Qwen2.5-7BCRACEOpenBookQA4850.257.47.2

W3

Following your suggestion, we added weight interpolation, LiNeS, and routing baselines on the same Qwen2.5-1.5B fine-tuned models to compare general recovery, domain retention, and application costs. Routing requires both pretrained and fine-tuned models, while direct LoRA switching requires a detachable adapter; full fine-tuning and checkpoints supplied only as merged weights do not support direct adapter switching. To test applicability in that setting, we also applied the existing HyperNet to two-epoch fully fine-tuned Qwen2.5-1.5B models, improving their general average from 50.03 to 54.96. Thus, HyperRepair offers a recovery-retention alternative to routing and interpolation and remains applicable when direct adapter switching is unavailable. The baseline comparisons and full fine-tuning results are included in Appendices H.4 and H.2 of the revised paper.

BackboneMethodGeneral15 score (%)Domain score (%)Additional preparation time (s)Decoding latency (ms/token)
Qwen2.5-1.5BBase54.1147.07052.69
Qwen2.5-1.5Bfine-tuning47.1585.030109.77
Qwen2.5-1.5BWeight interpolation53.9085.780.000036110.67
Qwen2.5-1.5BLiNeS50.6986.220.000033110.70
Qwen2.5-1.5BRouting53.9585.50601.31120.56
Qwen2.5-1.5BHyperRepair54.50 ± 0.1185.51 ± 0.005.51116.38

W4

Following your suggestion, we added a Qwen2.5-1.5B experiment in which RSNs and HyperNet are trained on only six general benchmarks. The remaining nine benchmarks are completely excluded from both RSN and HyperNet training and used only for evaluation. On these unseen benchmarks, HyperRepair raises the average from 55.69 to 59.66 ± 0.14, compared with 58.81 for Base. Thus, the results show that HyperRepair generalizes to general benchmarks unseen by both repar-training stages. We have included these results in Appendix H.1 of the revised paper.

BackboneTrained or held outEvaluated general benchmarkBaseFine-tunedHyperRepair(mean ± std)
Qwen2.5-1.5Bheld outARC-Challenge44.9742.6350.31 ± 0.10
Qwen2.5-1.5BtrainedHellaSwag67.8462.8376.18 ± 0.15
Qwen2.5-1.5BtrainedMMLU59.7034.9452.42 ± 0.75
Qwen2.5-1.5Bheld outOpenBookQA40.2040.2745.82 ± 0.80
Qwen2.5-1.5Bheld outPIQA75.9573.6377.03 ± 0.09
Qwen2.5-1.5Bheld outSciQ93.3088.4793.31 ± 0.20
Qwen2.5-1.5Bheld outWinoGrande63.6964.9367.20 ± 0.17
Qwen2.5-1.5BtrainedANLI-R139.5032.0042.49 ± 1.31
Qwen2.5-1.5Bheld outCOPA83.0082.6784.89 ± 0.19
Qwen2.5-1.5BtrainedSocial IQa49.1347.4449.95 ± 0.16
Qwen2.5-1.5Bheld outLogiQA31.0330.4731.78 ± 0.06
Qwen2.5-1.5Bheld outWiC52.9849.6351.76 ± 0.49
Qwen2.5-1.5BtrainedMATH28.361.8319.07 ± 0.10
Qwen2.5-1.5BtrainedHumanEval37.8027.0337.63 ± 0.41
Qwen2.5-1.5Bheld outMBPP44.2028.5334.84 ± 0.63
Qwen2.5-1.5Btrainedtrained 6 general benchmark average score47.0534.3546.29 ± 0.37
Qwen2.5-1.5Bheld outheld out 9 general benchmark average score58.8155.6959.66 ± 0.14
Qwen2.5-1.5Ball15 general benchmark average score54.1147.1554.31 ± 0.22

Q1

Yes. In the added Qwen2.5-1.5B experiment (See additional experimental table in W4), all nine evaluation benchmarks are completely excluded from both RSN and HyperNet training; both components are trained on six other general benchmarks. HyperRepair improves all nine benchmark averages, raising the aggregate score from 55.69 to 59.66 ± 0.14 (Base: 58.81). Thus, HyperRepair can recover general performance on benchmarks that neither RSNs nor HyperNet encountered during repair training. We have included these results in Appendix H.1 of the revised paper.

Q2

We tested the adaptation changes you ask about on Qwen2.5-1.5B: LoRA rank 16/two epochs, rank 64/two epochs, rank 64/four epochs, and two-epoch full fine-tuning. We kept the learned HyperNet fixed rather than retraining it for each target configuration. General performance improves in all four settings, including 54.27 to 56.95 for rank 16/two epochs, 45.12 to 51.86 for rank 64/four epochs, and 50.03 to 54.96 ± 0.10 for full fine-tuning. The results show that a fixed HyperNet can improve recovery across the tested LoRA ranks, fine-tuning durations, and full fine-tuning setting without configuration-specific retraining. We have included these results in Appendix H.2 of the revised paper.

BenchmarkBaseLoRA rank16/epoch2LoRA rank16/epoch2 + HyperRepairLoRA rank64/epoch2LoRA rank64/epoch2 + HyperRepairLoRA rank64/epoch4LoRA rank64/epoch4 + HyperRepairFull Fine-tuning epoch2Full fine-tuning epoch2 + HyperRepair
ARC-Challenge44.9748.5554.5542.6351.71 ± 0.3242.3249.5242.9251.36 ± 0.22
HellaSwag67.8467.0976.1262.8373.56 ± 0.3461.4171.9663.9174.86 ± 0.36
MMLU59.7059.0658.3034.9452.53 ± 0.5828.5041.2845.8751.75 ± 0.54
OpenBookQA40.2040.6045.8040.2746.49 ± 0.1940.0046.2741.6046.87 ± 0.71
PIQA75.9575.9777.9373.6377.14 ± 0.3573.1475.9374.0877.48 ± 0.21
SciQ93.3095.4396.3088.4794.12 ± 0.4087.8093.1391.8394.53 ± 0.19
WinoGrande63.6963.7768.1464.9368.34 ± 0.1362.9065.8064.6168.91 ± 0.14
ANLI-R139.5035.0045.4332.0042.69 ± 0.2530.9040.1731.4041.22 ± 0.31
COPA83.0085.6785.6782.6785.33 ± 1.4583.6784.0082.6785.11 ± 1.50
Social IQa49.1350.0553.2847.4450.44 ± 0.0646.6749.1847.8350.34 ± 0.33
LogiQA31.0332.1032.7230.4731.23 ± 0.1828.7829.8530.2630.67 ± 0.59
WiC52.9854.1351.9349.6351.86 ± 0.9849.9551.2049.5851.41 ± 0.24
MATH28.3627.2728.301.8320.38 ± 0.270.0320.6315.1124.96 ± 0.24
HumanEval37.8037.2037.5227.0333.08 ± 1.0020.9330.9332.3233.20 ± 1.00
MBPP44.2042.1342.2028.5338.60 ± 0.6419.8728.0736.5341.80 ± 0.33
Average54.1154.2756.9547.1554.50 ± 0.1145.1251.8650.0354.96 ± 0.10

Q3

To answer your cost comparison, we added Qwen2.5-1.5B measurements for offline HyperNet training, new-target preparation, and decoding, alongside interpolation, LiNeS, and routing. The General15 HyperNet averages 3.17 GPU-hours per seed, after which preparing a new target's repair takes 5.51 seconds without further optimization. For comparison, routing takes 601.31 seconds of preparation and 120.56 ms/token, versus 5.51 seconds and 116.38 ms/token for HyperRepair. This separates the cost of learning a reusable repair mapping from the cost incurred for each new target. Thus, the trained HyperNet can be reused across targets, with lower preparation time and decoding latency than the tested router. The preparation-time and decoding-latency comparisons are included in Appendix H.4 of the revised paper.

BackboneMethodGeneral15 score (%)Domain score (%)Additional preparation time (s)Decoding latency (ms/token)
Qwen2.5-1.5BBase54.1147.07052.69
Qwen2.5-1.5Bfine-tuning47.1585.030109.77
Qwen2.5-1.5BWeight interpolation53.9085.780.000036110.67
Qwen2.5-1.5BLiNeS50.6986.220.000033110.70
Qwen2.5-1.5BRouting53.9585.50601.31120.56
Qwen2.5-1.5BHyperRepair54.50 ± 0.1185.51 ± 0.005.51116.38

Comments on the paper

1 comment
Area_Chair_PU2mOpenReview ↗

Author Reviewer Discussion

Dear reviewers,

Thanks again for agreeing to review for ICLR.

As we are approaching the end of the discussion period, I would appreciate your acknowledgment of the authors' rebuttal and any feedback you would like to share with the authors.

This will help to improve the submissions, streamline decision making, and keep ICLR's original idea of having such a discussion period valuable to all participants in the review process.

Best, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.