- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes HyperRepair for restoring general-task performance of fine-tuned language models while retaining domain specialization. It trains gated Repairing Side Nets (RSNs) for source models, then learns a hypernetwork that generates repair parameters from activation fingerprints. For a model adapted to a held-out domain, repair generation requires neither target-domain examples nor additional optimization. Experiments evaluate two backbones across twelve adaptation domains and fifteen general benchmarks, with ablations examining fingerprint conditioning, functional supervision, and selective gating.
Critical strengths
- The paper addresses an important problem: domain specialization can degrade general capabilities, while the original adaptation data may no longer be available. Generating a repair without target-domain optimization is a useful objective. The frozen-backbone design and prompt-conditioned correction provide a coherent approach to balancing general recovery with domain retention.
- HyperRepair improves the average general score of all six held-out models in Table 1, while Figure 3 shows that domain performance is largely retained. The shared-repair and always-on-gate ablations provide useful evidence for model-specific conditioning and selective correction. These results make the approach promising despite the uneven recovery across individual benchmarks.
Critical weaknesses
- Table 1 supports partial, task-dependent recovery rather than consistently restored capabilities. For Qwen adapted to SNIPS, MBPP reaches only 23.2 versus the base model’s 63.8; for Qwen adapted to DROP, repair reduces MBPP from 58.8 to 56.8. The headline average can therefore mask substantial residual deficits and regressions. Category-level summaries, base-relative gaps, and multiple-seed results would clarify robustness.
- As a minor presentation issue, Table 2’s dense wrapping and equation placement make the three-way comparison harder to scan; clearer column spacing and response separation would help.
Questions for authors
Appendix Table 5 shows that repair training uses labeled data from the evaluated general-task families, including proxy data for code generation. This warrants distinguishing supervised general-task learning from recovery that transfers beyond those families. Figure 4 already includes a shared-RSN control, but its aggregate result on one backbone and split offers limited insight into where conditioning helps.
4 comments
Q1
To examine recovery beyond the benchmarks used for general-task supervision, we added a Qwen2.5-1.5B experiment training RSNs and HyperNet on six general benchmarks and completely excluding nine others from both training stages. On these nine unseen benchmarks, the average improves from 55.69 to 59.66 ± 0.14. We distinguish this benchmark-level holdout from task-family novelty: HumanEval and MBPP, for example, both evaluate code generation. Thus, these results demonstrate generalization to benchmarks unseen by both repair-training stages, with task-family overlap explicitly distinguished. We have included these results in Appendix H.1 of the revised paper.
| Backbone | Trained or held out | Evaluated general benchmark | Base | Fine-tuned | HyperRepair(mean ± std) |
|---|---|---|---|---|---|
| Qwen2.5-1.5B | held out | ARC-Challenge | 44.97 | 42.63 | 50.31 ± 0.10 |
| Qwen2.5-1.5B | trained | HellaSwag | 67.84 | 62.83 | 76.18 ± 0.15 |
| Qwen2.5-1.5B | trained | MMLU | 59.70 | 34.94 | 52.42 ± 0.75 |
| Qwen2.5-1.5B | held out | OpenBookQA | 40.20 | 40.27 | 45.82 ± 0.80 |
| Qwen2.5-1.5B | held out | PIQA | 75.95 | 73.63 | 77.03 ± 0.09 |
| Qwen2.5-1.5B | held out | SciQ | 93.30 | 88.47 | 93.31 ± 0.20 |
| Qwen2.5-1.5B | held out | WinoGrande | 63.69 | 64.93 | 67.20 ± 0.17 |
| Qwen2.5-1.5B | trained | ANLI-R1 | 39.50 | 32.00 | 42.49 ± 1.31 |
| Qwen2.5-1.5B | held out | COPA | 83.00 | 82.67 | 84.89 ± 0.19 |
| Qwen2.5-1.5B | trained | Social IQA | 49.13 | 47.44 | 49.95 ± 0.16 |
| Qwen2.5-1.5B | held out | LogiQA | 31.03 | 30.47 | 31.78 ± 0.06 |
| Qwen2.5-1.5B | held out | WiC | 52.98 | 49.63 | 51.76 ± 0.49 |
| Qwen2.5-1.5B | trained | MATH | 28.36 | 1.83 | 19.07 ± 0.10 |
| Qwen2.5-1.5B | trained | HumanEval | 37.80 | 27.03 | 37.63 ± 0.41 |
| Qwen2.5-1.5B | held out | MBPP | 44.20 | 28.53 | 34.84 ± 0.63 |
| Qwen2.5-1.5B | trained | trained 6 general benchmark average score | 47.05 | 34.35 | 46.29 ± 0.37 |
| Qwen2.5-1.5B | held out | held out 9 general benchmark average score | 58.81 | 55.69 | 59.66 ± 0.14 |
| Qwen2.5-1.5B | all | 15 general benchmark average score | 54.11 | 47.15 | 54.31 ± 0.22 |
Q2
To show where conditioning helps beyond the aggregate ablation, we added shared-versus-conditioned RSN comparisons on split D for SNIPS-adapted Llama3 and LEDGAR-adapted Qwen2.5-1.5B. The shared control retains HyperNet training and RSN generation, so this comparison isolates conditioning on the target model. Conditioning raises MBPP from 0 to 16.8 for Llama3 and MMLU from 35.12 to 53.11 for Qwen2.5-1.5B, showing which adaptation-task pairs benefit from conditioning beyond the shared repair. Thus, conditioning improves repair for specific adaptation-task pairs across the two evaluated backbones, beyond the shared-RSN control.
| Backbone | Split | domain task | general task | fine-tuned | w/o fingerprint | HyperRepair | Conditioning benifits |
|---|---|---|---|---|---|---|---|
| Llama3.1-8B | D | SNIPS-7 | MBPP | 1.2 | 0 | 16.8 | 16.8 |
| Llama3.1-8B | D | SNIPS-7 | MATH | 0 | 0 | 9.4 | 9.4 |
| Qwen2.5-1.5B | D | LEDGAR | MMLU | 24.59 | 35.12 | 53.11 | 17.99 |
| Qwen2.5-7B | A | HateXplain | COPA | 75 | 85 | 92 | 7 |
| Qwen2.5-7B | C | RACE | OpenBookQA | 48 | 50.2 | 57.4 | 7.2 |
Thank you for the additional experiments and the revisions to Table 2. The task-level, multi-seed results clarify where recovery is consistent and where substantial gaps remain. The gains on benchmarks excluded from both repair-training stages strengthen the case for transfer to unseen benchmarks, and the shared-RSN comparison shows that fingerprint conditioning can help on specific task pairs. This evidence addresses my main concerns sufficiently for me to raise my rating from 2 to 3.
Dear Reviewer Pmn2,
Thank you for taking the time to review our additional experiments and revised manuscript, and for your positive reassessment. Your suggestions helped us present a more precise account of both the benefits and limitations of HyperRepair.
We appreciate your thoughtful and constructive engagement throughout the discussion.
Best,
Authors
We thank the reviewer Pmn2 for the valuable feedback, which has helped us identify opportunities to improve the clarity and rigor of our work.
W1 To check whether averages hide inconsistent recovery, we added task-level, multi-seed results on Qwen2.5-1.5B with rank-64/two-epoch LoRA and full fine-tuning, reporting fine-tuned, repaired, and Base scores. In the LoRA setting, MBPP improves from 28.53 to 38.60 ± 0.64, remaining 5.60 points below Base at 44.20, while the general average rises from 47.15 to 54.50 ± 0.11. We will summarize these results by capability and discuss your original SNIPS→MBPP deficit of 40.6 points and DROP→MBPP regression of 2.0 points separately, keeping the evidence from the two backbones distinct. Thus, the results support aggregate recovery in the tested settings while identifying residual task-level deficits and individual regressions. The additional Qwen2.5-1.5B results are included in Appendix H.2 of the revised paper.
W2
Following your suggestion, we have revised Table 2 in the updated PDF by increasing column spacing, improving text wrapping, and more clearly separating the outputs of the base model, the fine-tuned model, and HyperRepair. We have also adjusted the placement of the formulas in the MATH example, making the three-way comparison easier to scan.