- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The motivation of this paper are potential mismatches between FFT and non-periodic weight matrices, and the limitations of using a single frequency cutoff across layers. It proposes DCT-based high-pass filtering with per-layer cutoffs determined by cross-task sign disagreement. Experiments across four model settings show that adjusting the update scale and task weights has a much larger effect than changing the transform or cutoff, with the final method performing comparably to DARE. The results therefore primarily support the importance of combination rules, instead of supporting the effectiveness of the proposed DCT and adaptive cutoffs.
Critical strengths
-
The evaluation covers multiple model families and sizes.
-
The authors report mixed results and identify potentially useful implementation confounds.
Critical weaknesses
- Insufficiently supported motivation. The paper does not establish why the properties of DCT should improve model merging, or why sign disagreement should determine low-frequency filtering strength.
- Incomplete experimental controls. The experiments do not adequately separate transform choice, cutoff adaptation, task weighting, and update magnitude.
- Limited evidence for the central contribution. Most gains arise from combination-rule adjustments, while the proposed methodological innovations show small and mixed effects.
- Incomplete evaluation and presentation. The unresolved math evaluation issue and unsupported claims weaken the manuscript’s completeness.
Questions for authors
-
The introduction motivates DCT partially by its energy-compaction properties for smooth signals. Are the task-vector matrices actually smooth along their rows or columns? Also, since the method removes low frequencies, concentrating more energy there could remove more useful information. What evidence suggests that DCT separates interference from useful task information better than FFT? Similarly, why should a layer with more sign disagreement need more low-frequency removal?
-
The paper compares FFT-FIXED, DCT-FIXED, and DCT-ADAPTIVE, but omits FFT-ADAPTIVE. Could you include this comparison? Could you also compare against a fixed cutoff equal to the average adaptive cutoff? Otherwise, it is unclear whether any gain comes from adapting the cutoff to each layer or simply changing the overall filtering strength.
-
Section 5.3 notes that uniform weighting also reduces the update norm. Does it still help when the two weighting schemes produce updates of the same norm? How does the final method compare with simply averaging the unfiltered task vectors using the same edit scale?
-
Section 4.1 attributes the math expert’s low score to the answer extractor failing to parse boxed answers. Could you fix the extractor and reevaluate both the expert and the merged models, rather than exclude math from the aggregate?
-
The first paragraph in the introduction describes model merging as a “standard building block” for combining domain-specialized models, but provides no references. Could you add references supporting this claim or use more qualified wording?