acceptodds
Under review as a conference paper at ICLR 2027

ReForge: Refining Merged Models with Anchor-Regularized Regression

Abstract

Model merging aims to combine multiple task-specific expert models into a single model without joint retraining, offering a practical alternative to multi-task learning when data access or computational budget is limited. Existing model merging methods rarely exploit strong merged models as priors for further improvement. To address this limitation, we propose ReForge, a bilevel optimization framework that formulates module-wise refinement as Bayesian linear regression with an anchor-centered prior. The inner level yields a closed-form MAP estimate from unlabeled calibration activations. The outer level uses Bayesian optimization to jointly select heterogeneous regularization strengths and assembly scales using held-out validation data. Furthermore, we develop a data-free variant of ReForge that replaces activation statistics with task-vector Grams, eliminating the need for calibration examples. Across extensive benchmarks, including up to 20-task merging in vision and 5-task merging in language, ReForge consistently outperforms all evaluated plug-and-play anchor baselines (e.g., TA, WUDI-Merging, and TSV). On 20-task ViT-B/32, ReForge improves the strongest evaluated baseline, ISO-CTS, from 77.6% to 82.8% in the data-assisted setting and to 81.5% in the data-free setting. On eight-task ViT-L/14, the data-assisted variant achieves 95.1% mean accuracy, compared with 95.8% for the individual task experts. Our source code is available at https://anonymous.4open.science/r/reforge-6CE5.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
model merging, task vectors, Bayesian linear regression, Bayesian optimization, bi-level optimization, data-free merging

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_v986OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes ReForge, a post-hoc refinement method for model merging. For each linear layer, it solves a ridge regression problem regularized toward an existing merged model (the anchor), using either calibration activations (ReForge-DA) or normalized task-vector Gram matrices (ReForge-DF). On top of this closed-form inner update, an outer Bayesian optimization loop tunes block- and group-wise regularization weights and scaling factors on a validation set. Experiments on ViT (up to 20 tasks) and Llama-3B/8B (5 tasks) show consistent gains over seven anchor baselines.

Critical strengths

Treating existing merged models as priors in a closed-form ridge regression step is clean and practical. It connects activation-based merging like RegMean, weight-Gram merging like WUDI, and existing subspace anchors under one framework. I also appreciate the control experiments in the appendix (replacing the task-vector Gram with random PSD or identity matrices, and testing pure anchor rescaling under the same BO budget), which clearly show the contribution of each component.

Critical weaknesses

ReForge-DF replaces iterative WUDI optimization with a ridge-regularized closed-form solve. It would be helpful to cite and discuss its connection to SWUDI (Closed-Form Spectral Regularization for Multi-Task Model Merging), which also studies the ill-posed WUDI normal equation and solves it in closed form via spectral filtering. In addition, the appendix ablations show that on strong anchors like ISO-CTS, rescaling the anchor via BO already provides a large portion of the gain, while the task-vector Gram adds a smaller margin (0.4% to 1.0%). Discussing this breakdown in the main text would give readers a clearer picture.

Since the outer BO loop evaluates candidates on a labeled validation set across all tasks, the term "data-free" for ReForge-DF (and 1-shot or 5-shot in the few-shot experiments) may confuse some readers, and it would help to clarify how validation data availability differs from calibration data in practice. Moreover, on weaker anchors (Pretrained, TA, TIES, RegMean) and on Llama-3.2-3B, ReForge-DF outperforms activation-based ReForge-DA.

Questions for authors

Could you discuss how the ridge regularization in ReForge-DF relates to the closed-form spectral filtering in SWUDI?

Why does ReForge-DF outperform ReForge-DA on weaker anchors and on Llama-3.2-3B, and could the validation inputs used in BO also serve as unlabeled calibration samples for the inner loop?

Could you briefly explain the choice of block numbers (B=1 for Llama vs. B=3 for ViTs), and why 20 BO trials on Llama-3.2-3B perform slightly below the initial TSV anchor?

5 comments
AuthorsOpenReview ↗

Response to Reviewer v986 (1/2)

(1/2)

Thank you for the positive assessment and the precise questions. We now cite SWUDI and discuss it as concurrent work.

Relation to SWUDI (Q1).

(a) How Eq. (14) relates to SWUDI. For one linear module, let U(t)=W(t)−WpreU^{(t)} = W^{(t)} - W_{\mathrm{pre}} be the task vector of expert tt, U(0)U^{(0)} the task vector of the anchor, and Bt=U(t)⊤U(t)/∥U(t)∥F2B_t = U^{(t)\top} U^{(t)} / \|U^{(t)}\|_F^2 the unit-trace Gram of U(t)U^{(t)} (Eq. 13). Let C=∑tBtC = \sum_t B_t and D=∑tU(t)BtD = \sum_t U^{(t)} B_t, and let C=∑kμkvkvk⊤C = \sum_k \mu_k v_k v_k^\top be the eigendecomposition of CC. The general estimator of SWUDI (its Eq. 11) is

τ^=τinit+(D−τinitC)Ch†,\hat\tau = \tau_{\mathrm{init}} + (D - \tau_{\mathrm{init}} C) C^{\dagger}_{h},

where τ^\hat\tau is the merged task vector, τinit\tau_{\mathrm{init}} is a base point, and Ch†=∑k(hk/μk)vkvk⊤C^{\dagger}_{h} = \sum_k (h_k/\mu_k) v_k v_k^\top applies a filter hkh_k to each nonzero eigenvalue μk\mu_k of CC. Choosing the Tikhonov filter hk=μk/(μk+λ~)h_k = \mu_k/(\mu_k + \tilde\lambda) with regularization strength λ~>0\tilde\lambda > 0 and the anchor as the base point, τinit=U(0)\tau_{\mathrm{init}} = U^{(0)}, gives

τ^=(D+λ~U(0))(C+λ~I)−1,\hat\tau = (D + \tilde\lambda U^{(0)})(C + \tilde\lambda I)^{-1},

which is our Eq. (14); the two also agree on the null space of CC, where DD vanishes and both keep the anchor component. The methods differ in what they design. SWUDI designs the filter, using a soft exponential filter with hard spectral truncation around a fixed base point. ReForge keeps the Tikhonov filter, uses any existing merged model as the base point, selects the regularization strength of each module group on validation data, and also has an activation-based variant, ReForge-CA. We derived this closed form independently: our preprint appeared on arXiv about three weeks before SWUDI, and we give the dates and the link to the Area Chair in a confidential comment (Section 4.3 and Appendix E).

(b) SWUDI as an anchor. Because ReForge accepts any merged model as its anchor, we ran SWUDI and its adaptive variant ASWUDI with the authors' code (unmodified) on the same fine-tuned experts as in all our experiments and applied ReForge on top (ViT-B/32, five seeds; change relative to the anchor in parentheses; Table 14):

AnchorTasksAnchor alone+ ReForge-CF+ ReForge-CA
SWUDI887.8987.88 (−0.01)89.70 (+1.81)
SWUDI1482.2482.58 (+0.34)84.80 (+2.56)
SWUDI2078.8979.37 (+0.48)82.91 (+4.02)
ASWUDI887.7287.84 (+0.12)89.69 (+1.97)
ASWUDI1482.1982.39 (+0.20)84.80 (+2.61)
ASWUDI2078.9879.42 (+0.44)82.75 (+3.77)

ReForge-CA improves both anchors by 1.8 to 4.0 points. ReForge-CF adds at most 0.5 points, as expected because it lies in the same filtering family as SWUDI.

Revision: Section 4.3 now states this relation and cites SWUDI as concurrent work; Appendix E gives the derivation and the SWUDI-anchor results (Table 14).

Rescaling breakdown. We agree and now report this breakdown in the main text (Section 5.3, Table 4). We compare ReForge with a rescaling control that uses the same 24 parameters and 200 BO evaluations but spends all of them on rescaling (see our response to Reviewer jmsx, W1(b)). The gain has two parts, and both come from ReForge. First, searching module-specific hyperparameters with BO is itself part of our design: rescaling alone raises the 20-task ISO-CTS anchor from 77.6% to 81.8%, and BO also outperforms a shared setting or random search (Table 3). Second, the regression adds a further gain on top of it: 1.6 points for ReForge-CF and 2.4 points for ReForge-CA in this setting, and overall ReForge-CA outperforms the rescaling control on all six anchors and ReForge-CF on five, all except WUDI.

Revision: this comparison replaces Table 7 of the original submission and is now in the main text (Section 5.3, Table 4), with ViT-L/14 results in Appendix B (Table 9).

"Data-free" terminology. We agree and, as you suggested, have renamed ReForge-DF and ReForge-DA to ReForge-CF (calibration-free) and ReForge-CA (calibration-assisted) throughout the revised paper and this response.

AuthorsOpenReview ↗

Response to Reviewer v986 (2/2)

(2/2)

(a) What the names mean. The new names only indicate whether calibration activations are used. Both variants select hyperparameters on labeled validation data, as do tuned baselines such as TSV and ISO-CTS, and "1-shot" and "5-shot" refer only to the number of unlabeled calibration examples per task. The abstract and introduction now state this, and Appendix J (Table 19) lists the validation-set sizes. The dependence on validation data is mild: on 20-task ViT-B/32, using only 10% of the validation set gives relatively stable results (Figure 3).

(b) Validation vs. calibration data in practice. A validation set only has to score a candidate model; in our Llama experiments it consists of small benchmark splits scored by task metrics such as IFEval prompt accuracy, GSM8k exact match and MBPP pass@1. A calibration set instead supplies unlabeled inputs that represent the experts' task domains, whose activations define the regression in every linear layer. A setting may therefore have a scored validation benchmark but no suitable inputs from the experts' domains, or the reverse; ReForge-CF requires only the former.

Revision: the variants are renamed ReForge-CF and ReForge-CA throughout the paper; the abstract and introduction state that both use labeled validation data for hyperparameter selection; Appendix J (Table 19) lists the validation-set sizes.

Why does ReForge-CF outperform ReForge-CA on weak anchors (Q2)? The same pattern appears in our new matched comparison: ReForge-CA is best on the strong anchors (ISO-CTS, TSV, WUDI), and ReForge-CF on the weak ones (RegMean, TA, TIES) (Table 4). A possible explanation is that a weak anchor is far from the experts and needs corrections in many input directions. With 128 calibration images per task, the activation Gram concentrates its energy in a few directions (on 20-task ViT-B/32, the top 10% of directions hold about 73% of its energy, versus 46% for the task-vector Gram), so it carries little information about the remaining directions, whereas the task-vector Gram spreads its information more evenly. In practice, the two variants are complementary: ReForge-CF suits weak anchors, while ReForge-CA suits strong ones.

Can validation inputs serve as calibration samples (Q2)? Yes. We used the unlabeled validation inputs as the calibration samples of ReForge-CA (Appendix F, Table 15). Test accuracy (%) of ReForge-CA on ViT-B/32, mean over five seeds:

AnchorCalibration set8 tasks14 tasks20 tasks
TAseparate (paper)84.9879.2075.54
TAvalidation inputs84.8979.3275.67
TSVseparate (paper)88.9084.3382.01
TSVvalidation inputs89.0384.3582.23
ISO-CTSseparate (paper)90.1585.4982.77
ISO-CTSvalidation inputs90.1485.4582.85

In all nine settings the two choices differ by less than 0.25 points, so ReForge-CA can reuse the validation set that hyperparameter selection already requires instead of a separate calibration set.

Revision: new Appendix F (Table 15) and a sentence in Section 5.4.

Block numbers (Q3). For ViT, B = 3 (15 search dimensions) was a design choice, and later results suggest that more hyperparameters can help further: the 24-dimensional ReForge of our new matched comparison is more accurate than the 15-dimensional one in Table 1, e.g., 83.3% versus 81.5% (ReForge-CF) on 20-task ViT-B/32 with ISO-CTS (Section 5.3, Table 4; response to Reviewer jmsx, W1(b)). For Llama, each validation evaluation is far more expensive, so we limit the search to K = 100 evaluations; to search sufficiently within this budget, we use B = 1, which gives 5 dimensions.

K = 20 on Llama-3.2-3B (Q3). With K = 20, ReForge reaches 0.465, slightly below the TSV anchor's 0.471 (Table 18; Table 10 of the original submission). Twenty evaluations in a five-dimensional space leave the search still exploring, and the configuration is selected on validation data, which does not guarantee a gain on the test set. For this reason, our main Llama experiments use a larger budget of K = 100 (Table 2).

Dear Reviewer v986, thank you again for your feedback. We would appreciate hearing whether our response and the revised paper have addressed your main concerns. If anything remains unclear, we would be happy to provide further clarification.

Reviewer_9CyfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ReForge presents a framework for refining existing merged models through anchor-regularized Bayesian regression and jointly optimized hyperparameters. The regression can be done in both data-free and data-aware manners. Empirical results demonstrate improvements over several merging baselines.

Critical strengths
  • Experiments show performance improvement over anchor methods when ReForge is applied on vision and text datasets. The ablation and analyses are thorough.
Critical weaknesses

Weak motivations. The main motivation for the paper is that most current methods do not leverage existing merged models as priors to further refine the merged solution. This is a rather weak motivation as it is not conceptually clear why this is a drawback of existing methods. What are the gains of doing so?

Limited novelty.

  • ReForge aims to align the residual outputs between the merged model and individual in the data-aware regime. This has been explored in existing works such as Representation Surgery or RegMean, which also optimizes an activation-based linear regression problem. This makes the residual alignment step an adaptation of existing methods. If this is the case, the authors should cite these works properly.
  • Viewing model merging as MAP inference under noise assumption has been considered in a recent work [3]. Eq (14) in the paper is essentially eq (12) in [3] with the covariance matrix BtB_t. This makes the data-free variant of ReForge an insatance of established Gaussian MAP-based merging methods.
  • Bayesian optimization to find optimal merging coefficients has been done in [4]. How does ReForge's BO step position against this?

Overall ReForge seems to me as a performance boosting mechanism for existing merging schemes, which combines theoretical components of other methods rather than a conceptual innovation for model merging.

Experiments

  • The blue numbers indicating improvements don't make sense to me. On ViT-B/32 8 tasks, the pretrained anchor performance is 47.7, the Reforge-DA version performance is 84.8, yet somehow the improvement in subscript is +77.8? This inconsistency is presence throughout the table.

[1] Yang, Enneng, et al. "Representation surgery for multi-task model merging." arXiv preprint arXiv:2402.02705 (2024).

[2] Jin, Xisen, et al. "Dataless knowledge fusion by merging weights of language models." arXiv preprint arXiv:2212.09849 (2022).

[3] Bui, Long Minh, et al. "Model Merging as Probabilistic Inference in Fine-Tuning Parameter Space." arXiv preprint arXiv:2607.01689 (2026).

[4] Liu, Deyuan, et al. "Maximizing Intermediate Checkpoint Value in LLM Pretraining with Bayesian Optimization." ICML. 2025.

Questions for authors

N/A.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 9Cyf

Thank you for the detailed comments and references. We have clarified the motivation, separated our contribution from established components, and added the missing citations and comparisons.

Motivation: why use a merged model as a prior, and what is gained?

We agree that the motivation in the introduction can be improved. The point is not that existing merging methods are flawed: they produce good merged models, and these models can be improved further. ReForge uses such a merged model as the prior and refines it with observed evidence from the experts, namely their activations (ReForge-CA) or their task vectors (ReForge-CF), to obtain a posterior distribution over the merged model. This prior-to-posterior refinement is the core novelty of ReForge, and it brings two gains:

  • Every evaluated anchor improves. ReForge improves all seven anchors on ViT-B/32, ViT-L/14, Llama-3.2-3B and Llama-3.1-8B (Tables 1 and 2).
  • A better prior gives a better result. With the same regression and search, replacing the pretrained model by the ISO-CTS merged model as the prior raises ReForge-CA from 75.6% to 82.8% on 20-task ViT-B/32 and from 86.0% to 91.5% on 20-task ViT-L/14 (Table 1); for ReForge-CF the effect is smaller (80.0% to 81.5% on ViT-B/32).

Revision: the abstract and the second and third paragraphs of the introduction now state this motivation.

Novelty.

Our contribution is a framework that refines any existing merged model by using it as the prior of a regression toward the experts, with group-wise regularization strengths selected on validation data. None of the three lines of work discussed below refines a given merged model in this way. Within this framework, we compare two sources of expert information, activation statistics (ReForge-CA) and task-vector Grams (ReForge-CF). We address the three related lines of work below.

(a) RegMean, Representation Surgery. As λ → 0, Eq. (8) reduces to a RegMean-type solution (Section 4.1), and RegMean is both a baseline and an anchor in our experiments. The difference is the prior: RegMean corresponds to a prior centered at the pretrained model, whereas ReForge centers it at an existing merged model, and with the same regression and search this raises ReForge-CA from 75.6% to 82.8% on 20-task ViT-B/32 (Table 1). For Representation Surgery, the two approaches are complementary: Surgery keeps the merged weights fixed and adds a module for each task at inference, whereas ReForge changes only the weights, and training Surgery on the ReForge-CF model instead of the anchor improves it in all 18 settings. Please see our response to Reviewer jmsx (W2) for the full comparison; the revised paper adds these results in Section 5.4 and Appendix D.

(b) Bui et al. [3]. Eq. (14) is a Gaussian MAP estimate of the form of their Eq. (12), with one additional term centered at the anchor (Section 4.3 and Appendix E). We derived this closed form independently: our preprint appeared on arXiv about seven weeks before [3], and we give the dates and the link to the Area Chair in a confidential comment, so [3] is concurrent work, which we now cite. The closed form is only one component of ReForge; our focus is on using a given merged model as the anchor and selecting the regularization strength of each module group.

(c) Liu et al. [4]. Liu et al. use BO to combine adjacent checkpoints along one LLM pretraining run. ReForge uses BO for a different problem, selecting the group-wise regularization strengths and block scales of a closed-form refinement of independently fine-tuned experts. Moreover, a single shared λ chosen by grid search, without BO, already improves every anchor evaluated in Table 3, so the gains do not depend only on BO.

Finally, ReForge is indeed designed to improve existing merging methods: it improves all seven evaluated anchors (Tables 1 and 2) and can be combined with representation-level methods such as Representation Surgery and ProbSurgery.

Revision: Section 4.1 now states the relation to RegMean below Eq. (8); Section 4.3 and Appendix E relate Eq. (14) to Bui et al. and SWUDI; Related Work cites Representation Surgery, ProbSurgery, Bui et al., SWUDI and Liu et al.

Blue numbers in Tables 1 and 2.

Thank you for pointing this out; we agree that our presentation was not clear. Each blue subscript is the relative improvement over the anchor, not the absolute gain. For example, (84.8 − 47.7) / 47.7 = +77.8%, which corresponds to an absolute gain of 37.1 percentage points. Because the caption stated this only briefly, the values could easily be read as absolute gains. Revision: we have clarified this in the revised paper. The captions of Tables 1 and 2 now give the formula, 100 × (ReForge − anchor) / anchor, with this example in Table 1, and Sections 1 and 5.2 state the gains in absolute percentage points (e.g., average gains of 14.7 and 11.9 points over TA and TIES).

Dear Reviewer 9Cyf, thank you again for your feedback. We would appreciate hearing whether our response and the revised paper have addressed your main concerns. If anything remains unclear, we would be happy to provide further clarification.

Thank the authors for the clarification and positioning against related work. My concerns have been addressed and I do not have further questions.

Thank you for your positive feedback and for raising your score. We are glad that our response and the revision have addressed your concerns, and we will keep the clarifications and the positioning against related work in the final version.

Reviewer_jmsxOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReForge, a framework that further improves models produced by existing model merging methods. ReForge uses the merged model as an anchor and refines its weights through Anchor-Regularized Bayesian Regression, which incorporates information from individual expert models. It also uses Bayesian Optimization to optimize the regularization strength and scaling factors for each module. The authors introduce two variants. ReForge-DA uses activations from unlabeled calibration data, while ReForge-DF replaces activation statistics with a Task-Vector Gram Matrix, allowing refinement without calibration data. However, both variants still require labeled validation data for hyperparameter optimization. ReForge generally improves the performance of the model merging methods evaluated in the paper. In particular, in the 20-task setting with ViT-B/32, it improves ISO-CTS accuracy from 77.6% to 82.8% (ReForge-DA) and 81.5% (ReForge-DF).

Critical strengths
  • ReForge demonstrates consistent performance improvements across various merging baselines, model architectures, and task configurations. In particular, it achieves meaningful gains in challenging 20-task merging scenarios, where maintaining performance across multiple tasks is difficult. The improvements are also observed on both vision and language benchmarks.
  • ReForge can refine models produced by existing merging methods without changing those methods. Its anchor-regularized regression has a closed-form solution and does not require gradient-based retraining.
Critical weaknesses

W1. It is unclear whether ReForge's performance improvements mainly come from the proposed method itself or from additional validation-guided optimization. As shown in Table 1, ReForge achieves larger improvements with relatively weaker anchors such as TA and TIES, while the gains are smaller with stronger baselines such as TSV, WUDI, and ISO-CTS [1]. Notably, ISO-CTS already performs 225 hyperparameter search trials, yet ReForge applies an additional 200 Bayesian optimization trials on top of this optimized anchor. Table 7 shows that validation-guided anchor rescaling improves accuracy from 77.6% to 80.5% on ViT-B/32 with 20 tasks. Although the two approaches use different parameterizations, ReForge-DF provides an additional gain of only about 1 percentage point over the rescaling control (80.5% to 81.5%), raising questions about the independent contribution of the proposed regression. Although Appendix B already matches the nominal refinement search dimension and the 200-evaluation budget, comparisons under matched total optimization budgets, including anchor construction, would help clarify ReForge's benefits beyond additional optimization.

W2. ReForge is proposed as a framework for further improving existing merged models. However, its experiments mainly compare against standalone model merging methods. Representation Surgery [2] and ProbSurgery [3] also aim to improve existing merged models by correcting representation bias. Although these methods differ from ReForge in their specific approaches, they share the same goal of improving model performance after merging. Therefore, direct comparisons on the shared image classification model merging benchmark using CLIP ViT-B/32 and ViT-L/14 would help better demonstrate ReForge's advantages over existing methods. Such comparisons should also consider differences in data requirements, additional parameters, and computational costs.

References [1] D. Marczak et al., "No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces," ICML, 2025 [2] E. Yang et al., "Representation Surgery for Multi-Task Model Merging," ICML, 2024 [3] Q. Wei et al., "Representation Surgery in Model Merging with Probabilistic Modeling," ICML, 2025

Questions for authors

Please refer to the concerns and questions raised in the Weaknesses section.

7 comments
AuthorsOpenReview ↗

Response to Reviewer jmsx (1/4)

(1/4)

Thank you for the careful review; below are our responses to W1 and W2.

W1. Does the gain come from the regression or from additional validation-guided optimization?

We agree that the comparisons in the submission did not separate the effect of the regression from the effect of additional validation-guided search. We added two controls: the first tests whether the gain comes from extra search, and the second whether it comes from a larger number of hyperparameters.

(a) Matched total budget, including anchor construction. The 225 trials in Table 10 of the original submission estimate the cost of searching the full ISO-CTS grid; the Table 1 anchor searched only α (with the common-space fraction fixed at its default of 0.8), so ReForge's 200 trials are not added on top of a 225-trial search. As a total-budget control, we gave each anchor method 200 extra validation evaluations of its own hyperparameters on a finer grid that contains the original configurations. ViT-B/32, test accuracy (%); the anchor columns come from a deterministic grid search, and the ReForge columns are the five-seed means ± standard deviations of Table 1, so they differ from those in (b):

TasksAnchorOriginal searchOwn search + 200ReForge-CF (200 evals)ReForge-CA (200 evals)
8TA70.5070.9286.99 ± 0.0584.98 ± 0.05
8TIES75.7675.5987.03 ± 0.0685.67 ± 0.03
8RegMean82.2982.7687.92 ± 0.0985.23 ± 0.05
8TSV85.6985.5787.76 ± 0.0488.90 ± 0.05
8WUDI86.9886.9287.58 ± 0.0489.44 ± 0.02
8ISO-CTS86.1686.2289.03 ± 0.0390.15 ± 0.03
14TA65.2465.4782.38 ± 0.0379.20 ± 0.04
14TIES68.2668.1082.28 ± 0.1079.67 ± 0.04
14RegMean76.5277.0882.89 ± 0.0979.09 ± 0.02
14TSV79.9479.9282.85 ± 0.0384.33 ± 0.04
14WUDI80.5080.4781.69 ± 0.0684.55 ± 0.04
14ISO-CTS81.3681.3784.34 ± 0.0585.49 ± 0.06
20TA60.4361.3078.39 ± 0.0675.54 ± 0.04
20TIES64.0564.5679.24 ± 0.0576.33 ± 0.03
20RegMean72.2773.0879.74 ± 0.0375.85 ± 0.06
20TSV76.8776.9279.68 ± 0.0582.01 ± 0.04
20WUDI76.0776.1678.05 ± 0.0282.20 ± 0.06
20ISO-CTS77.5777.6281.52 ± 0.0482.77 ± 0.05

Across 8, 14 and 20 tasks, the 200 extra evaluations change the anchors by −0.2 to +0.9 points, while ReForge-CF exceeds these extended anchors by 0.7 to 17.1 points (Appendix C, Table 10).

AuthorsOpenReview ↗

Response to Reviewer jmsx (2/4)

(2/4)

(b) Regression vs. rescaling. The rescaling comparison in Table 7 of the original submission was not matched in search dimension or ranges. In the new control, both searches use 24 parameters, 200 Bayesian-optimization trials, the same validation set and the same three initial configurations, with block scales fixed to 1. The control uses all 24 parameters as scaling factors on groups of the anchor's task vector, whereas ReForge uses 12 scaling factors and 12 regularization strengths of the regression. ViT-B/32 (Section 5.3, Table 4), test accuracy (%), mean ± std over five seeds:

TasksAnchorRescaleReForge-CFReForge-CA
8TA78.62 ± 0.0988.17 ± 0.1286.27 ± 0.05
8TIES81.66 ± 0.1688.12 ± 0.0486.74 ± 0.05
8RegMean85.76 ± 0.0788.97 ± 0.0786.62 ± 0.06
8TSV87.76 ± 0.0688.80 ± 0.0989.30 ± 0.06
8WUDI88.54 ± 0.0788.54 ± 0.2289.80 ± 0.08
8ISO-CTS87.54 ± 0.0989.52 ± 0.0290.33 ± 0.07
14TA71.00 ± 0.0883.77 ± 0.1380.50 ± 0.09
14TIES75.43 ± 0.0883.66 ± 0.1881.01 ± 0.08
14RegMean80.45 ± 0.1284.28 ± 0.1580.59 ± 0.16
14TSV82.84 ± 0.1084.33 ± 0.1885.09 ± 0.10
14WUDI83.95 ± 0.0883.70 ± 0.1285.03 ± 0.08
14ISO-CTS83.95 ± 0.1085.45 ± 0.2286.32 ± 0.04
20TA67.07 ± 0.0680.59 ± 0.1677.89 ± 0.05
20TIES71.37 ± 0.0381.26 ± 0.1278.67 ± 0.06
20RegMean78.08 ± 0.1381.80 ± 0.0578.38 ± 0.07
20TSV81.10 ± 0.0782.02 ± 0.2383.11 ± 0.10
20WUDI81.52 ± 0.1581.03 ± 0.1183.06 ± 0.08
20ISO-CTS81.77 ± 0.0883.32 ± 0.1784.15 ± 0.10

On 20-task ISO-CTS, the setting the reviewer cites, ReForge-CF is 1.6 points and ReForge-CA 2.4 points above the control. Across 8, 14 and 20 tasks, ReForge-CA exceeds the control on all six anchors (on RegMean by only 0.1 to 0.9 points), ReForge-CF exceeds it on five (all except WUDI), and the better variant per anchor leads by 1.1 to 13.5 points. ViT-L/14 shows the same pattern (Appendix B, Table 9). ViT-L/14, mean ± std over five seeds:

TasksAnchorRescaleReForge-CFReForge-CA
8TA90.80 ± 0.0694.40 ± 0.1193.42 ± 0.04
8TIES91.34 ± 0.0594.32 ± 0.0993.80 ± 0.06
8RegMean93.05 ± 0.0994.41 ± 0.0993.47 ± 0.05
8TSV94.07 ± 0.0494.51 ± 0.0694.67 ± 0.02
8WUDI94.51 ± 0.1294.57 ± 0.0894.78 ± 0.11
8ISO-CTS94.76 ± 0.0994.97 ± 0.0795.19 ± 0.06
14TA85.09 ± 0.0691.17 ± 0.0789.87 ± 0.09
14TIES84.78 ± 0.0991.18 ± 0.1989.95 ± 0.03
14RegMean89.28 ± 0.0591.31 ± 0.1189.84 ± 0.02
14TSV90.87 ± 0.1091.62 ± 0.0991.95 ± 0.13
14WUDI91.48 ± 0.0591.34 ± 0.0892.07 ± 0.08
14ISO-CTS91.50 ± 0.1192.30 ± 0.0792.53 ± 0.05
20TA79.14 ± 0.0390.21 ± 0.1588.36 ± 0.03
20TIES82.19 ± 0.1190.24 ± 0.1088.40 ± 0.03
20RegMean87.42 ± 0.0590.52 ± 0.0988.35 ± 0.05
20TSV90.40 ± 0.0690.96 ± 0.0691.55 ± 0.08
20WUDI90.64 ± 0.0790.46 ± 0.0591.28 ± 0.03
20ISO-CTS91.10 ± 0.1491.60 ± 0.1191.99 ± 0.04

On ViT-L/14, ReForge-CF exceeds the control in 16 of the 18 configurations (all except WUDI with 14 and 20 tasks) and ReForge-CA in all 18.

Note that searching module-specific hyperparameters with BO is itself one of our contributions: the 24-parameter rescaling control already improves the anchors, e.g., from 77.6% to 81.8% on 20-task ISO-CTS, and the regression adds a further gain on top of it. Together, the two controls show that the gains do not come from additional search per se, but from the anchor-regularized regression and the module-wise BO search.

Revision: the new Appendix C (Table 10) reports the total-budget control with +200 evaluations; Section 5.3 (Table 4) replaces Table 7 of the original submission with the matched 24-parameter control; and Appendix B (Table 9) adds ViT-L/14 results over five seeds.

AuthorsOpenReview ↗

Response to Reviewer jmsx (3/4)

(3/4)

W2. Comparison with Representation Surgery and ProbSurgery.

We agree that Representation Surgery and ProbSurgery share the goal of improving an existing merged model and should be compared. Section 5.4 and Appendix D compare them with ReForge using their official code and the same experts, anchors and benchmarks on CLIP ViT-B/32 and ViT-L/14. For a fair comparison, we train the Surgery modules only on the calibration images and validation inputs that ReForge-CA also uses, rather than on the test inputs used by their official protocol; Table 11 also reports the official protocol, which differs by 0.1–2.6 points.

(a) Accuracy: combination and direct comparison. The two kinds of methods are complementary: training Representation Surgery or ProbSurgery on top of ReForge-CF instead of the original anchor improves their accuracy in all 36 settings, and the highest accuracy in every setting among methods not trained on test inputs comes from this combination (Table 6; Table 11). Test accuracy (%), mean ± standard deviation over five seeds; bold: best overall; italic: best stand-alone method:

ModelAnchorTasksSurgeryProbSurgeryReForge-CFReForge-CASurg. on CFProb. on CF
ViT-B/32TA881.38 ± 0.1283.30 ± 0.0586.99 ± 0.0584.98 ± 0.0588.81 ± 0.0289.37 ± 0.04
ViT-B/32TA2078.97 ± 0.1280.66 ± 0.0978.39 ± 0.0675.54 ± 0.0484.16 ± 0.0185.29 ± 0.05
ViT-B/32TSV888.01 ± 0.0488.69 ± 0.0587.76 ± 0.0488.90 ± 0.0589.35 ± 0.0589.75 ± 0.06
ViT-B/32TSV2083.71 ± 0.0984.82 ± 0.0479.68 ± 0.0582.01 ± 0.0485.05 ± 0.0885.90 ± 0.05
ViT-B/32ISO-CTS889.12 ± 0.0589.50 ± 0.0689.03 ± 0.0390.15 ± 0.0390.26 ± 0.0390.56 ± 0.03
ViT-B/32ISO-CTS2085.06 ± 0.0285.67 ± 0.0581.52 ± 0.0482.77 ± 0.0586.15 ± 0.0486.77 ± 0.04
ViT-L/14TA889.32 ± 0.1190.91 ± 0.0494.20 ± 0.0192.78 ± 0.0594.63 ± 0.0794.76 ± 0.03
ViT-L/14TA2086.72 ± 0.0388.70 ± 0.0489.59 ± 0.0186.15 ± 0.0391.70 ± 0.0192.31 ± 0.01
ViT-L/14TSV893.70 ± 0.0394.00 ± 0.0394.35 ± 0.0194.45 ± 0.0394.73 ± 0.0394.88 ± 0.02
ViT-L/14TSV2090.65 ± 0.0591.40 ± 0.0390.29 ± 0.0290.85 ± 0.0392.06 ± 0.0192.55 ± 0.02
ViT-L/14ISO-CTS895.10 ± 0.0195.22 ± 0.0294.96 ± 0.0295.12 ± 0.0395.21 ± 0.0295.35 ± 0.03
ViT-L/14ISO-CTS2092.43 ± 0.0392.79 ± 0.0291.12 ± 0.1191.51 ± 0.0192.69 ± 0.0692.98 ± 0.05

In direct comparison, each side wins in different regimes: at least one ReForge variant is more accurate than per-task ProbSurgery in five of six 8-task configurations, whereas ProbSurgery is more accurate in five of six 20-task configurations (Table 11). The advantage of Surgery and ProbSurgery with many tasks comes from changing the model architecture: both train a new module for each task and keep it at inference, which adds trainable parameters that grow with the number of tasks (on 20 tasks, about 0.33M for Surgery and 3.9M for ProbSurgery on ViT-B/32, and 0.49M and 5.9M on ViT-L/14), whereas ReForge keeps the architecture and adds no parameters. When a single module is shared by all tasks, at least one ReForge variant outperforms shared-module Surgery in all 18 settings and shared-module ProbSurgery in 15 of 18 (Table 11).

Shared-module variants (one module for all tasks, trained on calibration and validation inputs), test accuracy (%), mean ± standard deviation over five seeds:

ModelAnchorTasksShared SurgeryShared ProbSurgeryReForge-CFReForge-CA
ViT-B/32TA873.90 ± 0.1579.97 ± 0.1086.99 ± 0.0584.98 ± 0.05
ViT-B/32TA2063.43 ± 0.1274.27 ± 0.1678.39 ± 0.0675.54 ± 0.04
ViT-B/32TSV886.62 ± 0.0487.62 ± 0.1087.76 ± 0.0488.90 ± 0.05
ViT-B/32TSV2078.73 ± 0.0681.52 ± 0.0879.68 ± 0.0582.01 ± 0.04
ViT-B/32ISO-CTS887.29 ± 0.0788.55 ± 0.0489.03 ± 0.0390.15 ± 0.03
ViT-B/32ISO-CTS2079.40 ± 0.1282.98 ± 0.0781.52 ± 0.0482.77 ± 0.05
ViT-L/14TA885.43 ± 0.1089.02 ± 0.0994.20 ± 0.0192.78 ± 0.05
ViT-L/14TA2075.01 ± 0.2083.90 ± 0.1589.59 ± 0.0186.15 ± 0.03
ViT-L/14TSV893.17 ± 0.0293.50 ± 0.0494.35 ± 0.0194.45 ± 0.03
ViT-L/14TSV2088.26 ± 0.0389.72 ± 0.0890.29 ± 0.0290.85 ± 0.03
ViT-L/14ISO-CTS894.90 ± 0.0195.00 ± 0.0294.96 ± 0.0295.12 ± 0.03
ViT-L/14ISO-CTS2090.74 ± 0.0391.81 ± 0.0391.12 ± 0.1191.51 ± 0.01

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: summary of revisions (to all reviewers)

We thank the reviewers for their careful reading and constructive feedback. Following Reviewer v986, we renamed ReForge-DF and ReForge-DA to ReForge-CF (calibration-free) and ReForge-CA (calibration-assisted), and we use these names throughout this response. These names distinguish whether calibration activations are required for the inner closed-form update; both variants use labeled validation data for hyperparameter selection. Following the reviewers' suggestions, we have updated the paper of our ICLR submission, and its main text is now one page longer. The revised paper marks all changes in blue. The main changes are:

  1. Source of the gains (Reviewers jmsx, v986). We added two controls. First, giving each anchor method 200 extra evaluations of its own hyperparameters changes its accuracy by less than one percentage point (Appendix C, Table 10). Second, Section 5.3 (Table 4) compares ReForge with a rescaling control. Both use 24 parameters, 200 BO trials and the same validation data; the control spends all 24 on scaling, whereas ReForge spends 12 on scaling and 12 on regularization. On ViT-B/32 with 8, 14 and 20 tasks (five seeds), ReForge-CA outperforms this control on all six anchors and ReForge-CF on five, all except WUDI. Note that searching module-specific hyperparameters with BO is itself one of our contributions, so the improvement of the 24-parameter rescaling control over the original anchors (e.g., from 77.6% to 81.8% on 20-task ISO-CTS) also reflects the benefit of this component. The gains therefore do not come from additional search, but from the two components of ReForge: the anchor-regularized regression and the module-wise BO search.
  2. Representation Surgery and ProbSurgery (Reviewers jmsx, 9Cyf). The two kinds of methods are complementary: training Representation Surgery or ProbSurgery on top of ReForge-CF instead of the original anchor improves their accuracy in all 36 settings and gives the highest accuracy in every setting among methods not trained on test inputs (Section 5.4, Tables 6 and 11). In direct comparison, each side wins in different regimes: ReForge is more accurate than per-task ProbSurgery in five of six 8-task configurations, whereas ProbSurgery is more accurate in five of six 20-task configurations (Table 11). The advantage of Surgery and ProbSurgery with many tasks comes from changing the model architecture: both train a new module for each task and keep it at inference, which adds trainable parameters that grow with the number of tasks (on 20-task ViT-B/32, about 0.33M for Surgery and 3.9M for ProbSurgery), whereas ReForge keeps the architecture and adds no parameters. With one module shared by all tasks, ReForge outperforms Surgery in all 18 settings (Table 11). In addition, ReForge needs fewer samples per task: with only 16 images per task on ViT-B/32, both Surgery variants fall below the anchor in all four settings, while ReForge-CF still improves it (Table 13).
  3. Relation to RegMean and concurrent work (Reviewers 9Cyf, v986). Section 4.1 shows that Eq. (8) reduces to a RegMean-type solution as λ → 0. Section 4.3 and Appendix E derive the exact relation of Eq. (14) to SWUDI and Bui et al. We derived this closed form independently: our preprint appeared on arXiv about three and seven weeks before them, respectively (dates and link in a confidential comment to the AC), so both are concurrent work. The closed form is only one component of ReForge, whose core is to refine any existing merged model as a prior, with group-wise regularization selected on validation data. ReForge-CA further improves SWUDI and ASWUDI by 1.8–4.0 percentage points when used as anchors (Table 14).
  4. Motivation and use of validation data (Reviewers 9Cyf, v986). We have improved how the abstract and introduction present our motivation. ReForge uses an existing merged model as the prior and refines it with observed evidence from the experts, namely their activations (ReForge-CA) or their task vectors (ReForge-CF), to obtain a posterior distribution over the merged model; this prior-to-posterior refinement is the core novelty of ReForge. It brings consistent gains: ReForge improves all seven evaluated anchors on vision and language benchmarks (Tables 1 and 2), and with the same regression and search, replacing the pretrained model by the ISO-CTS merged model as the prior raises ReForge-CA from 75.6% to 82.8% on 20-task ViT-B/32 (Table 1). Both variants select hyperparameters on labeled validation data, just as tuned baselines such as TSV and ISO-CTS do, so ReForge and these baselines are compared with access to the same data. Appendix J (Table 19) now lists the validation-set sizes; on 20-task ViT-B/32, using only 10% of this data leaves the results relatively stable (Figure 3).

These revisions strengthen the evidence for anchor-based refinement and clarify its relationship to other methods and its data requirements. The main text remains within the 10-page limit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.