acceptodds
Under review as a conference paper at ICLR 2027

Adaptive DCT-Domain Spectral Merging: Rethinking Frequency-Domain Model Merging

Abstract

Model merging combines task-specific fine-tunes of a shared base into one model without retraining or training data. Frequency-domain methods such as FR-Merging reduce cross-task interference by applying a fixed-cutoff 2D FFT high-pass filter to each task vector before combining them. We revisit this recipe along two axes: replacing the FFT with the discrete cosine transform (DCT), which better matches the real-valued, non-periodic structure of weight matrices, and replacing the global cutoff with a per-layer adaptive cutoff driven by a cheap, data-free cross-task sign-interference score. On MergeBench, however, we find that two combination-rule pitfalls dominate the transform choice: an uncalibrated edit-scale that produces larger updates than the baselines at matched layers, and energy-proportional weighting under which each task's contribution scales with the square of its task-vector norm, starving smaller-update domains (instruction-following receives under 6% of the combination weight versus an equal 20% share). Correcting both raises aggregate normalized performance from 0.806 to 0.966 on Llama-3.1-8B-Instruct, matching DARE (0.965) on the four domains we aggregate. A replication across two model families at four scales (Llama-3.1-8B, Llama-3.2-3B, Gemma-2-2B/9B; 36 spectral merges, 144 domain evaluations) confirms the combination rules as the dominant effect: edit-scale calibration lifts pooled four-domain scores by +0.11 on Llama-3.1-8B and +0.07 on Gemma-2-9B and is neutral to slightly negative on the two smaller bases, versus at most 0.015 for any DCT-vs-FFT contrast; the DCT is ahead in most pairwise comparisons, but by margins we do not claim as significant. We release the corrected method, the diagnostics, and comparisons against Task Arithmetic, TIES, DARE, and an FR-Merging-style FFT baseline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
model merging, task arithmetic, frequency-domain merging, discrete cosine transform, DCT, FFT, spectral filtering, task vectors, interference, MergeBench, large language models, multi-task model combination

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_TwbKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The motivation of this paper are potential mismatches between FFT and non-periodic weight matrices, and the limitations of using a single frequency cutoff across layers. It proposes DCT-based high-pass filtering with per-layer cutoffs determined by cross-task sign disagreement. Experiments across four model settings show that adjusting the update scale and task weights has a much larger effect than changing the transform or cutoff, with the final method performing comparably to DARE. The results therefore primarily support the importance of combination rules, instead of supporting the effectiveness of the proposed DCT and adaptive cutoffs.

Critical strengths
  1. The evaluation covers multiple model families and sizes.

  2. The authors report mixed results and identify potentially useful implementation confounds.

Critical weaknesses
  1. Insufficiently supported motivation. The paper does not establish why the properties of DCT should improve model merging, or why sign disagreement should determine low-frequency filtering strength.
  2. Incomplete experimental controls. The experiments do not adequately separate transform choice, cutoff adaptation, task weighting, and update magnitude.
  3. Limited evidence for the central contribution. Most gains arise from combination-rule adjustments, while the proposed methodological innovations show small and mixed effects.
  4. Incomplete evaluation and presentation. The unresolved math evaluation issue and unsupported claims weaken the manuscript’s completeness.
Questions for authors
  1. The introduction motivates DCT partially by its energy-compaction properties for smooth signals. Are the task-vector matrices actually smooth along their rows or columns? Also, since the method removes low frequencies, concentrating more energy there could remove more useful information. What evidence suggests that DCT separates interference from useful task information better than FFT? Similarly, why should a layer with more sign disagreement need more low-frequency removal?

  2. The paper compares FFT-FIXED, DCT-FIXED, and DCT-ADAPTIVE, but omits FFT-ADAPTIVE. Could you include this comparison? Could you also compare against a fixed cutoff equal to the average adaptive cutoff? Otherwise, it is unclear whether any gain comes from adapting the cutoff to each layer or simply changing the overall filtering strength.

  3. Section 5.3 notes that uniform weighting also reduces the update norm. Does it still help when the two weighting schemes produce updates of the same norm? How does the final method compare with simply averaging the unfiltered task vectors using the same edit scale?

  4. Section 4.1 attributes the math expert’s low score to the answer extractor failing to parse boxed answers. Could you fix the extractor and reevaluate both the expert and the merged models, rather than exclude math from the aggregate?

  5. The first paragraph in the introduction describes model merging as a “standard building block” for combining domain-specialized models, but provides no references. Could you add references supporting this claim or use more qualified wording?

Reviewer_XC7EOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes DCT-Adaptive, a frequency-domain model merging method based on the Discrete Cosine Transform (DCT). Building upon the existing FREE-Merging framework, the method replaces FFT with DCT to transform task vectors into the frequency domain and adaptively adjusts the high-pass filtering cutoff for each layer based on cross-task sign disagreement to reduce task interference. Additionally, the method incorporates sparse corrections to preserve important parameter updates and combines task vectors using a calibrated edit scale and equal weighting.

Experiments are conducted on MergeBench across multiple model scales from the Llama and Gemma families. The results show that DCT-Adaptive achieves performance comparable to baselines such as DARE, while highlighting the substantial impact of edit-scale calibration and task weighting on merging performance.

Critical strengths
  1. The paper explores DCT-based frequency-domain model merging and introduces an adaptive cutoff strategy to account for differences in task interference across layers.

  2. The paper provides useful empirical observations on the impact of edit-scale calibration and task weighting, which may help improve the design of future model merging methods.

Critical weaknesses
  1. The paper mainly builds upon FREE-Merging by replacing FFT with DCT and introducing an adaptive cutoff strategy, offering relatively limited methodological novelty. The motivation for choosing DCT is not sufficiently justified, and DCT-Fixed does not demonstrate consistent performance improvements.

  2. The experimental evaluation is insufficient. Compared with the original FREE-Merging work, the paper lacks vision experiments and direct comparisons with the full method, as well as key ablations such as FFT-Adaptive. Therefore, the current experimental evidence is insufficient to fully demonstrate the effectiveness of the proposed modifications.

  3. The experimental comparisons could be further strengthened. The results show that most performance gains come from Edit-scale calibration and Equal Weighting, while the additional benefits of DCT and Adaptive Cutoff are relatively limited. Furthermore, the methods are not evaluated under a unified hyperparameter search protocol, and Norm-matched controls are missing, making it difficult to clearly distinguish the contributions of conventional merging adjustments from those of the core method.

Questions for authors
  1. Could the authors include representative experimental settings from the original FREE-Merging work, particularly detailed vision experiments and comparisons with a broader range of model merging methods, to provide a more comprehensive evaluation of the proposed method?

  2. Why is DCT more suitable than FFT for reducing task interference in model merging? Given that DCT-Fixed does not show a clear advantage over FFT-Fixed in Table 1, could the authors provide further theoretical justification or empirical analysis?

  3. Could the authors include FFT-Adaptive experiments to disentangle the individual contributions of DCT and Adaptive Cutoff and further validate the necessity of replacing FFT with DCT?

  4. The experimental results suggest that most performance gains come from Edit-scale calibration and Equal Weighting rather than DCT or Adaptive Cutoff. Additionally, the methods are not evaluated under a unified hyperparameter search protocol. Could the authors provide more consistent hyperparameter comparisons and Norm-matched controls to better distinguish the benefits of conventional merging adjustments from those of the proposed method?

Reviewer_fKVbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper revisits spectral merging of task-specific fine-tunes that share a base model. It replaces a 2D FFT high-pass filter with an orthonormal 2D DCT and sets a per-tensor cutoff from the average sign disagreement of five task vectors. It also adds each vector's largest 1% of entries back as a static sparse correction. During development, the authors found two choices more consequential than the transform: using an edit scale of 1.0 for spectral merges while strong non-spectral baselines used 0.3, and weighting each filtered task vector in proportion to its norm, which gives small-update domains very little influence. They match the edit scale and use equal domain weights in the final method.

Critical strengths

The diagnostic work is careful and practical. The paper checks actual per-layer update norms instead of assuming that a nominal edit-scale hyperparameter makes methods comparable. It also measures the average combination weights over 224 tensors: instruction-following receives 0.056 and multilingual only 0.020 under norm-proportional weighting, compared with 0.20 each under equal weights. This provides a concrete explanation for why a seemingly reasonable spectral combination can suppress smaller-update domains. The staged ablation is useful: on Llama-3.1-8B, changing the edit scale raises DCT-ADAPTIVE from 0.806 to 0.945, while equal weighting raises it further to 0.966.

The authors are appropriately candid about what these numbers do and do not establish. They report a replication on four model bases, note that the scale fix is model dependent, state that DCT improvements are too small for significance claims, and disclose that their FFT baseline is a static FR-MERGING-style reimplementation rather than the full routed FREE-MERGING framework. This makes the paper a credible cautionary study of evaluation choices in merging, even if the proposed DCT method is not independently compelling.

Critical weaknesses
  1. The central adaptive-DCT claim is weakly isolated. DCT-ADAPTIVE reaches 0.966, DCT-FIXED 0.962 and FFT-FIXED 0.961 on the four-domain aggregate; those differences are smaller than plausible evaluation noise, and the paper runs one seed per configuration. It never evaluates FFT-ADAPTIVE, so it cannot separate the effect of changing the transform from the effect of changing the cutoff. More fundamentally, the sign-disagreement statistic does not demonstrate that cross-task conflict resides in low spatial frequencies of a weight tensor. Since hidden units can be permuted, “frequency” across matrix coordinates is itself a fragile heuristic. The main novelty therefore remains a proposed recipe rather than an experimentally supported explanation for performance.

  2. The headline performance remains sensitive to comparison and evaluation choices. Matching spectral methods to a fixed edit scale of 0.3 is helpful on Llama-3.1-8B and Gemma-2-9B, but reduces pooled raw scores by 0.003 on Llama-3.2-3B and 0.012 on Gemma-2-2B. The authors correctly conclude that the scale should be chosen per base, yet offer no data-free selection rule under their stated merging setting. Equal weighting may improve instruction following partly because it shrinks the total edit, a confound the paper itself acknowledges without a norm-matched control. Excluding math is understandable given the extractor mismatch, but it leaves the headline average over only four of the five merged domains; the proposed method has the lowest raw math score among the listed methods (0.798 versus 0.809–0.829). A repaired extractor could change the overall assessment.

Questions for authors
  1. Can the authors add an FFT-ADAPTIVE arm and a norm-matched comparison of equal versus norm-proportional weighting? Those two controls would determine whether adaptivity, the transform or merely a smaller merged update accounts for the remaining differences.

  2. Could the authors rerun the GSM8K evaluation with an answer extractor that handles the math expert's boxed notation, then include math in the normalized aggregate? This would show whether the current four-domain tie with DARE persists across the full benchmark. I would also like to see uncertainty across evaluation samples or repeated merging seeds for the sub-0.005 final differences.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.