acceptodds
Under review as a conference paper at ICLR 2027

Conflict-Guided Data-Free Model Merging

Abstract

Data-free model merging fuses several fine-tuned models into a single network. Recently proposed data-free methods estimate task statistics from model weights and apply regularization to reduce the interference between the fine-tuned specialized models. However, these approaches typically apply uniform regularization strength across layers, even though expert disagreement varies across the network. We introduce conflict-guided model merging, a data-free model merging scheme that assigns regularization strength to each layer based on a conflict score derived from the task vectors. This score quantifies how well a single shared update can accommodate competing experts. The conflict score guides both Ridge regularization and an orthogonalized iterative solver, while maintaining the existing task-statistics estimator. Experimental results show that regularization alone improves merging accuracy, and further gains come from adapting regularization strength to layer-specific conflict. Conflict-guided Ridge improves accuracy over ACTMat on CLIP ViT-B/32, with statistically significant gains when merging 14 and 20 tasks. For input-adaptive merging, selective storage has accuracy close to ours MASS baseline with  50% of its per-task factor storage.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
model merging, data-free model merging, task vectors, conflict-aware regularization, layer-wise interference, ridge regularization, Muon optimizer

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_S4HDOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper builds on ACTMat, a data-free RegMean variant that uses task-vector Gram matrices as a proxy for input covariances. It adds a per-layer "conflict score": the ACTMat loss at its closed-form optimum divided by the loss of the trivial zero merge, so the score lies between 0 and 1. After standardizing the scores across layers, the score sets either (i) a per-layer Ridge damping strength that grows exponentially with the layer's conflict, or (ii) a per-layer step size for a fixed-budget Muon-style solver initialized at the mean task vector.

Critical strengths

Careful attribution of where the gain comes from. Table 1 reports the regularization effect (U vs. ACTMat) and the conflict effect (C−UC - U) separately. Table 9 then holds the solver, damping law and pool fixed and changes only the per-layer score. The permutation control (20 draws per cell, with 0 of 20 reaching the true assignment in all four cells, App. A) shows that which layer gets which damping matters, not only the spread of λℓ\lambda_\ell. The replication on independent experts (App. G) found that the task-vector-norm alternative is unstable, and the authors removed a recommendation as a result. This matters because many merging papers give a single headline number; here the reader can see what is actually being claimed.

Critical weaknesses
  1. The contribution is narrow, the tuning comparison is uneven, and the dynamic part adds little. The method is a per-layer scaling of a Tikhonov term on ACTMat; ACE-Merging already damps a closed form of this objective with a single constant. The gains hold only for fully fine-tuned CLIP ViT-B and vanish on ViT-L/14 (Table 1), LoRA (Table 10) and T5 (Table 18), as the paper itself notes (L451-452). The comparisons behind "conflict is the signal" favour the conflict variant: the uniform-Ridge baseline reuses the strength tuned for the conflict schedule, the alternative descriptors in Table 2 reuse the conflict-tuned damping law, and external baselines run at published defaults (e.g., Task Arithmetic at 49.04% on B/16, T=14T=14). In the dynamic setting, the only comparison is the authors' own MASS port, which sometimes falls below the static merge (ViT-L/14 T=8T=8: 91.6 vs. 94.2, Table 3), and a simple "store the MLP projections" rule matches the conflict ranking (Table 15).

  2. The claims are systematically overstated, and the presentation is careless. The abstract reports "statistically significant gains when merging 14 and 20 tasks" on ViT-B/32, but Table 8 gives Holm-adjusted p=0.111p=0.111 (Ridge) and 0.2500.250 (Muon) at 14 tasks. Figure 1 headlines "More than 2× the uniform ridge's gain" using only the 8-task ViT-B Ridge cells, whose own Table 8 p-values (0.052 and 0.336) are not significant, while the same Table 1 shows no gain on ViT-L/14 and a negative Muon C−UC-U at B/16, T=14T=14; the figure also does not say whether its conflict heatmap and λ\lambda bars (panels 2-3) are real data or schematic. The figures are not of submission quality: in Fig. 3 (left) the legend sits on top of the MLP-in curve, the panel carrying the main evidence, and the right panel's y-axis label collides with the left panel. The text has many errors ("ours MASS baseline" in the abstract, "haven performance" L97, "We minimizes" L233, "didn't compared" L158-159, a sign error at L370-371), and the appendix is largely narrative. Together with Weakness 1, the paper overstates what the evidence supports and is not ready for publication.

Questions for authors
  1. Was the uniform-Ridge α\alpha (and the uniform-Muon η0\eta_0) selected on its own validation set with κ=0\kappa=0, or reused from the conflict variant's joint selection? Please report the best uniform α\alpha on a finer grid, selected on validation and evaluated on test, for all ViT-B cells. If the C−UC - U gain shrinks below about 0.5 points, my assessment of the main claim changes.
  2. What is the wall-clock and memory cost of the method compared with ACTMat and TSV, in particular of 500 Muon steps per layer and of the per-input factor-space re-merge (App. J)? The paper mentions per-job wall-clock in its manifest but reports no numbers.
  3. Are panels 2-3 of Figure 1 (per-layer conflict heatmap and λ\lambda bars from ×100\times100 to ×0.04\times0.04) computed from the actual ViT-B experts, or schematic? If real, which backbone and task set; if schematic, please label them as such.
3 comments
AuthorsOpenReview ↗

Response to reviewer: Additional experiments and clarifications

We thank the reviewer for the detailed assessment and for recognizing our careful attribution of the gains. Conflict-guided Ridge outperforms independently tuned uniform Ridge by +0.77 to +1.15 points across all six ViT-B settings. All six point estimates exceed your 0.5-point threshold, and four gains are significant (Holm-adjusted p < 0.05).

Q1/W1: Uniform Ridge. Uniform Ridge tunes on validation (18 values, 1e-5–1e-1 at 8 tasks; 14 values, 1e-4–1e-1 at 14/20), then tests. Conflict Ridge remains fixed (α=5e-4, κ=1).

8 tasks14 tasks20 tasks
ViT-B/16+1.02 (+0.94)+0.90 (+0.89)+0.77 (+0.76)
ViT-B/32+1.15 (+1.39)+1.07 (+1.52)+1.13 (+1.23)

C-U gains (original in brackets). Holm p (B/16): .040, .009, <.001; (B/32): .051, .051, .016. Tasks won: B/16 7/8, 11/14, 17/20; B/32 6/8, 12/14, 16/20. Largest drop: 0.45 (B/32, 14 tasks); all three B/16 gains held or rose. Muon moves to the appendix as a secondary solver.

W1: Descriptor controls. Each descriptor (conflict, type, depth, width, update norm) tunes κ, α over 18 settings (κ ∈ {−1, −0.5, 0.25, 0.5, 1, 2}, α ∈ {2.5e-4, 5e-4, 1e-3}). Conflict leads all six ViT-B settings (+0.20 to +0.55); five nominally significant, none after Holm (min adjusted p=.071). It beats all 20 score shuffles per setting (p=.048, minimum with 20 draws).

W1: Tuned baselines (TA was 49.04%). Each baseline tunes on its published validation grid: TA 0.1–1.0, TIES 0.8–2.5, TSV-M/Iso-C 0.25–3.0, WUDI four variants, ACE eps 1e-7–1e-2. TA rises 49.0→70.6 (B/16, 14), still 14 points below ACTMat. Fixed conflict Ridge beats all tuned baselines except Iso-C at B/16, 8 tasks (+1.55 vs. +1.52 over ACTMat).

W1: Beyond ACTMat. Conflict guidance complements ACE's tuned global damping, improving ACE without re-tuning in all six ViT-B settings (+0.64 to +1.06). Applied to each method's tuned baseline, it also improves Iso-C (+2.04; 6/6, p<.0001) and TSV-M (+0.44; p=.039). Shuffling removes the gain, so the gain depends on which layers are damped. RegMean (real input data) does not benefit (−0.9 to −1.1); we will report this.

W1: Generalization limits. Claims focus on fully fine-tuned ViT-B. On ViT-L/14 (8 tasks), a label-fitted schedule gains +0.06 over uniform (CI −0.05 to +0.17), vs. +0.03 for conflict; Whisper-small is similar. LoRA conflict is near zero (mean 0.0004 vs. 0.167 for full fine-tuning). No significant aggregate gain outside ViT-B: ViT-L/14 +0.07 to +0.13 (3 settings); Whisper mean +0.23 (CI −0.07 to +0.54, 18 settings); T5 mean +0.21 (CI −0.16 to +0.58, 10 settings). See Reviewer vj1z (W4).

W1: Dynamic merging. We remove our MASS reimplementation/comparisons. The 91.6 vs. 94.2 gap (L/14, 8) mixes true-head static and routed-head dynamic; official MASS reproduces 90.60 vs. 90.6 (B/16, 8). Whole-layer selection matches random/MLP (Table 15). At 5–10% of MASS storage, within-layer conflict selection beats random by 2.4–3.1 points and all 10 draws (p=.09, minimum with 10), matching MASS truncation; we report this in the appendix.

W2: Claims/presentation. We drop abstract 14-task significance (Holm p=.080 for 18 printed tests; .051 vs. tuned uniform; .111 came from 27 tests including an unprinted row). Figure 1 replaces “more than 2×” with Ridge gains vs. tuned uniform in all nine ViT settings (+0.77 to +1.15 on ViT-B, +0.07 to +0.13 on ViT-L/14). We fix Figure 3 labels, typos/sign wording, and appendix tables.

Q2: Cost. Merging is a one-time offline step; inference cost is unchanged.

MethodViT-B/16, 8 tasksViT-L/14, 20 tasks
ACTMat21 s131 s
TSV-M17 s122 s
Conflict Ridge22 s126 s
Muon (500 steps)47 s221 s

Reported merge-only times are medians of 3 idle-GPU runs. Ridge matches ACTMat peak memory (±0.02 GB): 0.7–1.8 GB (B/16), 1.3–3.2 GB (L/14) for 8–20 tasks. The one-off score computation takes ~11 s (B/16), ~60 s (L/14). The table gives Muon at ~2.1× and ~1.8× the Ridge times, respectively. MASS per-input re-merge (App. J) is removed.

Q3: Figure 1 provenance. Panels 2–3 show real data, not schematic: panel 2 plots r_ℓ for B/16 attention/MLP matrices over 8 tasks (12 blocks × 4); panel 3 plots 10^(κz_ℓ) for 12 MLP-input projections, ×0.04 (block 11) to ×100 (block 3). Panel 4 plots uniform/conflict Ridge gains vs. ACTMat at 8 tasks. Caption will clarify.

We thank the reviewer again; We believe the changes following the rebuttal have helped us improve the paper (which we are still updating), and we hope these improvements are reflected in your final assessment.

Thank you for the time and effort you have dedicated to reviewing our paper. As the discussion period is ending soon, we kindly ask you to look at the new results and clarifications in our rebuttal. We believe they address your main concerns, and we hope you will take them into account in your final assessment. We have also updated the manuscript to reflect these changes. If any concerns remain, we would be happy to provide further explanations.

Reviewer_S4HDOpenReview ↗

Follow-up on the revised comparisons and claims

Thank you for the detailed response and revision. The revised statistical claims, figure provenance, and total-cost reporting also resolve the corresponding issues in my review.

The revised controls address several important concerns. Although I still find the contribution limited, I will raise my rating from 1 to 2.

Reviewer_B29kOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a data-free rule for allocating regularization across layers. It retains the ACTMat objective and covariance proxy, and defines conflict as the optimal unregularized residual normalized by the error of retaining pretrained weights. The score sets Ridge strength or the Muon step size under a fixed iteration budget, and selects expert layers retained for dynamic merging. The main results concern fully fine-tuned CLIP ViT-B experts; limitations in other settings are also reported.

Critical strengths
  1. Expert disagreement varies across layers, making uniform regularization potentially inappropriate. Using the merging objective’s optimal residual to allocate regularization addresses this issue without task examples or a complex new module. The change is modest but addresses a concrete problem.
  2. Sections 5.2–5.3 and Appendix compare uniform allocation, alternative descriptors, shuffled scores, and a second expert set. These controls help distinguish improvements from the solver itself from those due to conflict-dependent allocation. I find this the strongest evidence in the paper.
Critical weaknesses
  1. The optimum residual measures incompatibility under the covariance proxy, but does not establish that shrinking toward a reference model will help. Ridge shrinks toward pretrained weights, whereas finite-budget Muon limits movement from the mean update. Since the objective, proxy, and solvers are inherited, the novelty largely rests on this mapping, whose conditions of validity remain insufficiently explained.
  2. The main dynamic comparison also changes the static backbone, making it difficult to attribute the overall gain to conflict-based selection. Table 15 compares selection criteria, whereas Table 16 measures the cost of half storage with a shared backbone; these answer different questions. The abstract’s significance statement for ViT-B/32 with 14 tasks also disagrees with the Holm-corrected results in Table 8. These claims should be narrowed accordingly.
Questions for authors
  1. Why should a larger proxy residual imply stronger shrinkage? How does this explanation cover both Ridge’s pretrained reference and Muon’s mean-update reference, and when would it fail?
  2. Does the residual score retain its advantage if uniform regularization and the strongest alternative descriptor each select their own damping parameters? Can the existing sweeps answer this?
  3. With the same static backbone and storage budget, how much does conflict-based layer selection itself contribute? Please explain using Tables 15–16 and reconcile the abstract’s ViT-B/32, 14-task significance claim with Table 8.
2 comments
AuthorsOpenReview ↗

Response to reviewer: Mechanism validation, additional controls, and clarifications

We thank the reviewer for the detailed assessment. A pre-specified test shows the conflict score predicts which layers benefit from damping (Spearman +0.67 / +0.56; +0.94 / +0.76 within MLP input projections) The data-free and label-informed schedules give similar test gains. Conflict Ridge also gains +0.77 to +1.15 points against independently tuned uniform Ridge across all six ViT-B settings.

W1 + Q1: Why should larger residuals imply stronger shrinkage? High conflict means no shared update fits all experts. Damping preserves pretrained weights where tasks disagree, while allowing agreed-upon updates elsewhere.

Direct mechanism test (fixed in advance). On ViT-B/16 and B/32, 8 tasks, we independently multiplied damping by 10 and 0.1 for each of 50 layers, starting from validation-tuned uniform Ridge. The difference in validation accuracy tracks conflict: Spearman +0.67 (B/16), +0.56 (B/32), permutation p < 10^-5; correlation with depth is only −0.2. Consistently, damping the 18 highest-conflict layers recovers 90% of the gain. Within MLP input projections, the correlations reach +0.94 / +0.76.

Comparison with label-informed schedules. A label-informed schedule tuned on the κ × α grid gains +0.88 and +1.01 over uniform on test (B/16 and B/32), vs. +0.94 and +1.15 for our data-free conflict schedule. On B/32, coordinate search across all 50 layers initialized from our schedule (~300 validation merges) raises validation by 0.30 but test accuracy by only +0.03 (CI −0.15 to +0.22).

W1 + Q1: Reference choice. Ridge shrunk toward pretrained weights, mean task vector, or TSV merge differs by at most 0.09 in six of seven settings and 0.26 in all (six ViT-B settings plus ViT-L/14; α = 5e-4). At 10× α, TSV improves by up to 0.9. Conflict exceeds uniform for every reference and strength (39 of 39; minimum +0.10 on ViT-L/14). Muon moves to the appendix; we have not tested its mean-update reference and will say so.

W1 + Q1: Failure cases. (1) No per-layer headroom: label-informed schedules fail to beat uniform on held-out ViT-L/14 (+0.06, CI −0.05 to +0.17) and Whisper (−0.33, CI −1.44 to +0.78); conflict also matches uniform. (2) Near-zero LoRA scores (mean 0.0004 vs. 0.167) collapse the schedule to uniform. (3) RegMean's real covariances: conflict costs 0.9 to 1.1 points. We will document all three.

Q2: Independently tuned controls. Uniform Ridge tunes on validation over 18 strengths (1e-5 to 1e-1; 8 tasks) or 14 (1e-4 to 1e-1; 14/20 tasks); conflict stays at α = 5e-4, κ = 1. Conflict gains +1.02, +0.90, +0.77 (B/16; 8, 14, 20 tasks), and +1.15, +1.07, +1.13 (B/32); Holm p = .040, .009, <.001, .051, .051, .016.

Conflict, layer type, depth, width, and update norm each tune κ and α from identical 18 combinations (κ ∈ {−1, −0.5, 0.25, 0.5, 1, 2}, α ∈ {2.5e-4, 5e-4, 1e-3}). Conflict ranks first in all six settings, by +0.20 to +0.55; nominally significant in five, not after Holm (smallest adjusted p = .071). It beats all 20 shuffled-score controls in each setting (p = .048, the minimum possible with 20 draws).

W2 + Q3: Dynamic merging and abstract. Table 15 compares selection rules with the same static backbone and storage budget. Whole-layer conflict selection shows no advantage over random or MLP selection. Table 16 measures the accuracy cost of halving storage with a shared backbone. Table 3 changes the backbone at the same time as the selection strategy, so it cannot isolate the effect of selection. We remove our MASS reimplementation and associated comparisons. Official MASS reproduces ViT-B/16, 8 tasks: 90.60 vs. 90.6. Within layers, the conflict score still carries a signal: at 5–10% of MASS's storage, conflict-selected components beat random ones by 2.4 to 3.1 points and exceed all 10 random draws (p = 0.09, the minimum with 10 draws), matching MASS's own truncation. We will add this to the appendix.

Beyond ACTMat. Conflict guidance complements ACE's tuned global damping, improving ACE without re-tuning in all six ViT-B settings (+0.64 to +1.06). Applied to each method's tuned baseline, it also improves Iso-C (+2.04; 6/6, p<.0001) and TSV-M (+0.44; p=.039). Shuffling removes the gain, so the gain depends on which layers are damped. RegMean (real input data) does not benefit (−0.9 to −1.1); we will report this.

We remove the abstract's 14-task significance claim: Holm-adjusted p = .080 over 18 printed tests; .051 versus tuned uniform. The reported 0.111 came from 27 tests, including an unprinted row. We instead give paired gains across all six ViT-B settings.

We thank the reviewer again; We believe the changes following the rebuttal have helped us improve the paper (which we are still updating), and we hope these improvements are reflected in your final assessment.

Thank you for the authors‘ response. I have no further questions and will maintain my original rating.

Reviewer_vJ1zOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper enhances the previous ACTMat algorithm for data-free model merging by introducing a layer-wise conflict score. This metric quantifies task interference by calculating the ratio of the best possible merged loss to the loss of simply retaining the original pretrained weights. Layers with high conflict scores are then automatically assigned stronger regularization (e.g., higher Ridge penalties) or smaller optimization steps (e.g., in Muon). On fully fine-tuned ViT-B models, conflict adds 0.76–1.52 points over uniform Ridge.

Critical strengths

It introducing a conflict-guided adaptive framework which applies stronger damping to high-conflict layers (acting as a safety brake to preserve pretrained knowledge) and weaker damping to low-conflict layers (allowing them to merge freely and retain expertise).

Critical weaknesses
  1. The method fully inherits ACTMat’s data-free covariance proxy and loss function. The "conflict score" mechanism is highly specific to this framework and cannot be easily transferred to other model merging paradigms. Furthermore, the empirical benefits appear largely confined to ViT architectures, limiting the method's broader impact.

  2. The proposed method's hyperparameters are carefully selected via validation sweeps, while external baselines are largely evaluated using their published defaults. For instance, Appendix F reveals that simply sweeping for a smaller λ\lambda in the absolute-Ridge baseline reverses a −2.96-point drop into a +0.24-point gain. Table 21 shows that the WUDI baseline's performance can be largely improved simply by changing its initialization from sum to mean. To ensure a fair comparison, all strong baselines must be provided with a matched validation-search budget.

  3. The abstract claims statistically significant Ridge gains over ACTMat on ViT-B/32 at both 14 and 20 tasks. However, according to the paper's own Holm-adjusted p-values in Table 8, the 14-task setting yields a p-value of 0.111 (> 0.05), which is not statistically significant.

  4. Most of experiments are conduct in ViT, the language and LoRA experiments yield mostly negative or unstable results. The unchanged Muon recipe loses 7–16 points on T5, and LoRA merging requires substantially different damping strategies (Table 11). These results suggest that the method's success relies heavily on architecture-specific heuristics.

  5. Muon adds 500 iterations with five Newton–Schulz steps, which maybe costy. The author should add concrete end-to-end efficiency analysis.

Questions for authors

See Weakness

5 comments

We thank the reviewer for the detailed assessment. The conflict schedule improves Iso-C (+2.04 points), TSV-M (+0.44), and ACE (+0.64 to +1.06) over their tuned baselines on ViT-B. Conflict-guided Ridge also gains +0.77 to +1.15 points across all six ViT-B settings against independently tuned uniform Ridge.

W1: Transfer beyond ACTMat. We hold ACTMat’s proxy/loss fixed to isolate the schedule. Each method uses its own validation-tuned baseline. Iso-C/TSV-M search 0.25–3.0 plus conflict-reachable scales; conflict searches 9 settings (scale ± neighbors, g ∈ {0.1, 0.2, 0.4}) using 10^(−g·z). ACE tunes eps 1e-7–1e-2; conflict reuses it with κ = 1, without re-tuning.

Across six ViT-B settings, all three methods improve overall: Iso-C +2.04 (6/6 settings, 71/84 tasks, p < .0001); TSV-M +0.44 (4/6, p = .039); ACE +0.64 to +1.06 (6/6, each p < .05 uncorrected). Score shuffling removes the gain (true minus shuffled: Iso-C +2.26; TSV-M +0.34, p = .003), supporting layer specificity. RegMean declines (−1.12 B/16, −0.91 B/32, 8 tasks); real covariances may already fit experts, so added damping hurts. We will report this limitation.

W2: Independently tuned baselines. Baselines tune on validation, then test. Uniform Ridge searches 18 strengths (1e-5–1e-1) for 8 tasks, 14 (1e-4–1e-1) for 14/20. TA/TIES/TSV-M/Iso-C use published grids; WUDI four variants (incl. Table 21 mean initialization); ACE eps 1e-7–1e-2. Conflict Ridge keeps the paper’s fixed setting (α = 5e-4, κ = 1) from the submission.

Conflict minus tuned uniform Ridge: B/16 +1.02, +0.90, +0.77; B/32 +1.15, +1.07, +1.13 (8/14/20 tasks; paired; Holm p = .040, .009, <.001, .051, .051, .016). Four gains survive Holm (p < 0.05). Conflict beats all tuned baselines except Iso-C on B/16, 8 tasks (+1.55 vs. our +1.52 over ACTMat). Elsewhere, ACE is strongest (+0.85–1.95 vs. our +1.04–2.21); WUDI trails 0.55–0.70; absolute-λ Ridge reaches +0.24 (App. F).

W3: Significance correction. Table 8's Holm family included an unprinted row (27 tests); for 18 printed tests, adjusted p = .080, vs. tuned uniform .051. We remove the 14-task claim.

W4: Language, LoRA, and T5. Claims focus on fully fine-tuned ViT-B; elsewhere conflict Ridge is not worse than a tuned uniform over tuned uniform: ViT-L/14 +0.07, +0.07, +0.13 (8, 14, 20 tasks); Whisper base/small/medium, 18 settings (5 to 20 tasks): ≥ uniform in 14, mean +0.23 (95% CI −0.07 to +0.54), largest loss −1.15; T5-small/base, 10 settings (7 to 15 tasks): ≥ uniform in 6, mean +0.21 (95% CI −0.16 to +0.58), largest loss −0.34. Selected Whisper-base/medium gain +1.0 to +1.4, T5-small up to +1.4; ViT-L/14, Whisper-small, T5-base roughly match uniform.

Protocol. Both tune α on held-out data: 32 held-out examples per task, drawn from the test split following the TIES/ACTMat convention and excluded from evaluation (seed 42, TIES/ACTMat protocol) and evaluates on the rest; T5 uses TIES/ACTMat validation and separate test. Uniform searches 14 strengths, conflict 5–14 on the same grid; scores are standardized by module family. κ was fixed on a first group of settings and reused unchanged on the rest, where mean gains are +0.12 (Whisper) and +0.18 (T5).

Why little per-layer headroom? Uniform Ridge improves ACTMat by +5.3 to +9.4 on Whisper-base (8–20 tasks). Even a label-fitted layer schedule fails to beat uniform held-out: Whisper-small, 20 tasks −0.33 (CI −1.44 to +0.78); ViT-L/14, 8 tasks +0.06 (CI −0.05 to +0.17). Muon/T5 losses move to appendix. LoRA conflict is near zero (0.0004 vs. 0.167 full tuning, B/16), reducing to uniform. ACTMat's low-rank proxy struggles: 45.1 vs. experts' 82.8 (B/16 LoRA).

W5: Offline cost. Model size and inference cost are unchanged. Merge-only medians (3 idle-GPU runs):

MethodViT-B/16, 8 tasksViT-L/14, 20 tasks
ACTMat21 s131 s
TSV-M17 s122 s
Conflict Ridge22 s126 s
Muon (500 steps)47 s221 s

One-off score computation takes ~11 s (B/16) and ~60 s (L/14), nearly task-count independent. Ridge matches ACTMat peak memory (within 0.02 GB): 0.7–1.8 GB on B/16, 1.3–3.2 GB on L/14 (8–20 tasks). In the table, Muon takes ~2.1× and ~1.8× the Ridge times, respectively.

Direct mechanism test. Layers benefiting most from damping have higher conflict (Spearman +0.67/+0.56, p < 10^-5). Label-based per-layer optimization raises validation, not test accuracy (+0.03, CI −0.15 to +0.22); see Reviewer B29k (Q1).

We thank the reviewer again; We believe the changes following the rebuttal have helped us improve the paper (which we are still updating), and we hope these improvements are reflected in your final assessment.

Thank you for the time and effort you have dedicated to reviewing our paper. As the discussion period is ending soon, we kindly ask you to look at the new results and clarifications in our rebuttal. We believe they address your main concerns, and we hope you will take them into account in your final assessment. We have also updated the manuscript to reflect these changes. If any concerns remain, we would be happy to provide further explanations.

I appreciate the authors' detailed response and additional experiments. The new Iso-C, TSV-M, and ACE results address my concern that the schedule is specific to ACTMat. The independently tuned baselines substantially address my concern about comparison fairness.

However, I still have the following reservations:

  • Muon’s computational cost relative to its benefit. The reported merge-only timings show that Muon takes approximately 1.8–2.1 times as long as conflict-guided Ridge. Given the comparable accuracy of the two variants in Table 1, the practical motivation for using the 500-step Muon solver remains insufficiently justified.
  • Marginal gains beyond fully fine-tuned ViT-B. I appreciate the additional Whisper, T5, and ViT-L/14 experiments. However, the gains remain marginal, and the reported 95% confidence intervals include zero. Thus my concern about the method’s limited applicability and broader impact remains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.