- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes Silent-Twin Subtraction (STS), a fine-tuning strategy that combines standard fine-tuning with silent fine-tuning. Specifically, STS subtracts a tunable proportion of the silent fine-tuning update from the standard fine-tuning update to improve cross-domain generalization. Experiments conducted on Qwen2.5-Omni-7B across several benchmarks, including MMAR, MMAU-mini, MMSU, and AIR-Bench, demonstrate that STS can outperform conventional fine-tuning.
Critical strengths
The observation that silent fine-tuning can also exploit textual shortcuts to improve in-domain accuracy is interesting and insightful. Furthermore, the proposed Silent-Twin Subtraction method provides a simple yet practically useful approach to mitigating shortcut learning and improving cross-domain generalization.
Critical weaknesses
-
Inconsistencies in experimental results. The reported experimental results appear inconsistent across different tables and figures. Specifically, the baseline results differ between Tables 1 and 2, although the discrepancies are relatively small (on the order of 0.0x). More importantly, Table 4 and Figure 5 report a maximum improvement of +2.2 on MMAR, whereas Table 1 reports a larger improvement of +3.4. These discrepancies raise concerns about the consistency and reproducibility of the experimental results. The authors should explain the inconsistencies in the experimental results.
-
Sensitivity to hyperparameter selection and limited practical robustness. Although STS demonstrates promising improvements, the optimal subtraction coefficient varies substantially across downstream datasets. This makes it difficult to identify a single configuration that consistently delivers substantial performance gains across different domains. In practical applications, exhaustive hyperparameter searches are often infeasible, particularly when labeled evaluation data from the target domain are unavailable. Moreover, selecting a suboptimal coefficient may even degrade performance. This sensitivity raises concerns about the robustness and practical applicability of the proposed method.
Questions for authors
-
Could the authors explain the inconsistencies in the experimental results reported across Tables 1, 2, and 4, as well as Figure 5? In particular, what accounts for the different MMAR improvements reported in Table 1 (+3.4) and Table 4/Figure 5 (+2.2)?
-
Could the authors report the performance of the STS checkpoint trained on AudioMCQ when evaluated on MMAU-mini, MMSU, and AIR-Bench?
-
Could the authors further validate the proposed method on tasks with more diverse output formats, rather than focusing primarily on multiple-choice question answering?
Response to Reviewer AQiz
All underlying runs are consistent and reproducible. The MMAR numbers in the initial draft measure three distinct comparisons within the same greedy evaluation batch:
Each value answers a distinct experimental question from the exact same model scores, so all three figures are mutually consistent.
Separately, the +3.9 pp figure in our question-level analysis uses a dedicated sampled readout with five samples per question at temperature 0.6. Tables 1 and 2 also come from separate evaluation batches, where MMAR Base scores are 61.58% and 61.56%. Every reported difference is computed against the Base run of its corresponding batch and protocol, and we now label each comparison explicitly.
STS controls adaptation through overall update magnitude β and Silent subtraction λ. Across four fixed-point transfer analyses, STS outperforms Normal fine-tuning in 12 of 16 off-selection comparisons, and all four negative point estimates occur under strong subtraction with λ≥1. Moderate subtraction reliably reduces transfer losses while preserving useful adaptation, though the exact peak varies by task.
To test selection without target-benchmark labels, we chose (β,λ)=(0.4,0.6) using only an AVQA source-domain subset. This source-selected pair improves over Normal across all four benchmarks: +1.82 pp on MMAR, +5.21 pp on MMAU-mini, +0.68 pp on MMSU, and +0.71 pp on AIR-Bench. We also evaluate Retention-matched Scaling as a simple practical reference on source validation data. Useful fixed settings are therefore attainable without target-specific tuning.
We evaluated fixed STS (0.4, 0.3) on AudioMCQ-StrongAC (2,112 audio-sensitive examples, 66 steps per seed) across MMAU-mini, MMSU, and AIR-Bench:
On AudioMCQ-StrongAC, STS tracks Scaling closely and sits slightly below Normal on MMAU-mini and AIR-Bench. By contrast, on HeteroAQA-40K (our 40,160-item AudioMCQ mixture), fixed STS (0.4, 0.3) improves over Normal by +3.76, +0.96, and +1.48 pp on the same three benchmarks. This comparison highlights a clear operating boundary: STS helps most when Normal training introduces negative transfer, whereas Normal training alone transfers well on audio-sensitive data.
We extended evaluation to the full MMAU-Pro benchmark (5,304 scored items), which includes 625 open-ended questions and 87 instruction-following items. Evaluating the 625 open-ended items directly tests free-form generation beyond multiple-choice scoring.
Normal fine-tuning drops by 3.00 pp relative to Base on open-ended generation, and STS recovers 2.23 pp of that loss. Both fine-tuning degradation and STS recovery therefore extend to open-ended answers. At the same time, STS (-0.76 pp) and Retention-matched Scaling (-0.33 pp) remain slightly below Base, indicating that the intervention primarily mitigates fine-tuning degradation on free-form generation.