acceptodds
Withdrawn from ICLR 2027

Learning Without Listening: Silent Twins for Weak-Evidence Audio-Language Fine-Tuning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Post-training of audio language models usually uses supervision with definite answers, yet the available sound may not sufficiently constrain these targets. In this setting, models may still obtain substantial in-domain gains from non-acoustic task signals. Using AVQA fine-tuning as an example, we observe that a large in-domain improvement does not consistently translate into cross-task gains; moreover, replacing all training audio with equal-duration silence still yields substantial in-domain gains. The matched silent twin thus provides a measurable reference for these training-induced changes. In our experiments, we find that normal and silent fine-tuning induce highly overlapping changes across multiple audio language models and transfer benchmarks, and both damage questions that the base model originally answered reliably. We propose STS (Silent-Twin Subtraction): it uses the weight update produced by silent fine-tuning as a reference, subtracts a tunable proportion of it from the normal fine-tuning update, and uses a second coefficient to control the overall magnitude of the combined update. Across models and benchmarks, silent twins reveal what fine-tuning learns without listening, and STS consistently outperforms conventional fine-tuning on every transfer benchmark we evaluate.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Audio-Language Models, Multimodal Learning, Post-Training, Fine-Tuning, Weak Evidence, Robust Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AQizOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes Silent-Twin Subtraction (STS), a fine-tuning strategy that combines standard fine-tuning with silent fine-tuning. Specifically, STS subtracts a tunable proportion of the silent fine-tuning update from the standard fine-tuning update to improve cross-domain generalization. Experiments conducted on Qwen2.5-Omni-7B across several benchmarks, including MMAR, MMAU-mini, MMSU, and AIR-Bench, demonstrate that STS can outperform conventional fine-tuning.

Critical strengths

The observation that silent fine-tuning can also exploit textual shortcuts to improve in-domain accuracy is interesting and insightful. Furthermore, the proposed Silent-Twin Subtraction method provides a simple yet practically useful approach to mitigating shortcut learning and improving cross-domain generalization.

Critical weaknesses
  1. Inconsistencies in experimental results. The reported experimental results appear inconsistent across different tables and figures. Specifically, the baseline results differ between Tables 1 and 2, although the discrepancies are relatively small (on the order of 0.0x). More importantly, Table 4 and Figure 5 report a maximum improvement of +2.2 on MMAR, whereas Table 1 reports a larger improvement of +3.4. These discrepancies raise concerns about the consistency and reproducibility of the experimental results. The authors should explain the inconsistencies in the experimental results.

  2. Sensitivity to hyperparameter selection and limited practical robustness. Although STS demonstrates promising improvements, the optimal subtraction coefficient varies substantially across downstream datasets. This makes it difficult to identify a single configuration that consistently delivers substantial performance gains across different domains. In practical applications, exhaustive hyperparameter searches are often infeasible, particularly when labeled evaluation data from the target domain are unavailable. Moreover, selecting a suboptimal coefficient may even degrade performance. This sensitivity raises concerns about the robustness and practical applicability of the proposed method.

Questions for authors
  1. Could the authors explain the inconsistencies in the experimental results reported across Tables 1, 2, and 4, as well as Figure 5? In particular, what accounts for the different MMAR improvements reported in Table 1 (+3.4) and Table 4/Figure 5 (+2.2)?

  2. Could the authors report the performance of the STS checkpoint trained on AudioMCQ when evaluated on MMAU-mini, MMSU, and AIR-Bench?

  3. Could the authors further validate the proposed method on tasks with more diverse output formats, rather than focusing primarily on multiple-choice question answering?

1 comment
AuthorsOpenReview ↗

Response to Reviewer AQiz

[W1, Q1] Exact reconciliation of reported results

All underlying runs are consistent and reproducible. The MMAR numbers in the initial draft measure three distinct comparisons within the same greedy evaluation batch:

Reported GainExact Comparison (Greedy Readout)
+3.41 ppNormal (61.47%) minus Silent Twin (58.06%)
+2.22 ppSTS (63.80%) minus Base (61.58%)
+2.33 ppSTS (63.80%) minus Normal (61.47%)

Each value answers a distinct experimental question from the exact same model scores, so all three figures are mutually consistent.

Separately, the +3.9 pp figure in our question-level analysis uses a dedicated sampled readout with five samples per question at temperature 0.6. Tables 1 and 2 also come from separate evaluation batches, where MMAR Base scores are 61.58% and 61.56%. Every reported difference is computed against the Base run of its corresponding batch and protocol, and we now label each comparison explicitly.

[W2] Hyperparameter sensitivity and practical use

STS controls adaptation through overall update magnitude β\beta and Silent subtraction λ\lambda. Across four fixed-point transfer analyses, STS outperforms Normal fine-tuning in 12 of 16 off-selection comparisons, and all four negative point estimates occur under strong subtraction with λ≥1\lambda \ge 1. Moderate subtraction reliably reduces transfer losses while preserving useful adaptation, though the exact peak varies by task.

To test selection without target-benchmark labels, we chose (β,λ)=(0.4,0.6)(\beta, \lambda) = (0.4, 0.6) using only an AVQA source-domain subset. This source-selected pair improves over Normal across all four benchmarks: +1.82 pp on MMAR, +5.21 pp on MMAU-mini, +0.68 pp on MMSU, and +0.71 pp on AIR-Bench. We also evaluate Retention-matched Scaling as a simple practical reference on source validation data. Useful fixed settings are therefore attainable without target-specific tuning.

[Q2] AudioMCQ-trained checkpoint transfer

We evaluated fixed STS (0.4, 0.3) on AudioMCQ-StrongAC (2,112 audio-sensitive examples, 66 steps per seed) across MMAU-mini, MMSU, and AIR-Bench:

MethodMMAU-mini (%)MMSU (%)AIR-Bench (%)
Base74.8463.6669.98
Normal76.0963.4371.46
STS (0.4, 0.3)75.9663.4570.78
Magnitude-matched Scaling75.9763.3770.69

On AudioMCQ-StrongAC, STS tracks Scaling closely and sits slightly below Normal on MMAU-mini and AIR-Bench. By contrast, on HeteroAQA-40K (our 40,160-item AudioMCQ mixture), fixed STS (0.4, 0.3) improves over Normal by +3.76, +0.96, and +1.48 pp on the same three benchmarks. This comparison highlights a clear operating boundary: STS helps most when Normal training introduces negative transfer, whereas Normal training alone transfers well on audio-sensitive data.

[Q3] Evaluation beyond multiple choice

We extended evaluation to the full MMAU-Pro benchmark (5,304 scored items), which includes 625 open-ended questions and 87 instruction-following items. Evaluating the 625 open-ended items directly tests free-form generation beyond multiple-choice scoring.

AVQA-Trained ModelOpen-Ended Change from Base (625 Items)
Normal-3.00 pp
Silent Twin-2.27 pp
STS (0.4, 0.3)-0.76 pp
Retention-matched Scaling-0.33 pp

Normal fine-tuning drops by 3.00 pp relative to Base on open-ended generation, and STS recovers 2.23 pp of that loss. Both fine-tuning degradation and STS recovery therefore extend to open-ended answers. At the same time, STS (-0.76 pp) and Retention-matched Scaling (-0.33 pp) remain slightly below Base, indicating that the intervention primarily mitigates fine-tuning degradation on free-form generation.

Reviewer_QMyMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper explores the phenomenon that audio/audio-visual language models can be improved by using text only data. The paper propose a silent twin which is trained on text-only data along a model trained on audio-text data.

Critical strengths

The silent-twin approach provides a simple, effective control for measuring how much audio-language fine-tuning learns with text only input or without meaningful audio input. The proposed method is easy and simple to implement with LoRA arithmetic.

Critical weaknesses

The proposed method is primarily evaluated using one model and one training dataset which limits the generality of the conclusions. STS also has limited performance improvement over the scaling baseline which does not hold for each benchmark. STS also requires tuning the scaling coefficients for each of the benchmarks which also highlights limited generality of the proposed method.

Questions for authors

The proposed method should be benchmarked on more datasets or datasets beyond AVQA on which text only input can achieve very high performance. The STS coefficients are selected separately for each transfer benchmark. How well does STS perform when coefficients are selected on one benchmark and applied to an unseen benchmark? Also given that some of the improvements over scaling are really small, are these results meaningfully different or statistically significant?

1 comment
AuthorsOpenReview ↗

Response to Reviewer QMyM

[Setup] Audio-preserving design of Silent Twins

Unlike text-only training that removes audio inputs, a Silent Twin replaces each training waveform with equal-duration silence. This design keeps the audio encoder and exact audio-token positions unchanged, and both Normal and Silent models are evaluated on real audio recordings.

In the same evaluation batch, a separate text-only control gains +9.26 pp on AVQA, closely tracking the Silent Twin's +9.40 pp gain. This close agreement confirms that equal-duration silence acts as a clean, format-preserving training control.

[W1, Q1] Generalization across datasets and models

We added two independent full training pools and an audio-sensitive contrast setting on Qwen2.5-Omni-7B.

Training PoolNormal Gain (pp)Silent Gain (pp)Silent Recovery
AVQA+11.95+9.4479%
MUSIC-AVQA+10.01+3.3233%
HeteroAQA-40K+14.16+13.3895%

Silent training recovers substantial in-domain gains across diverse datasets, and the recovery rate varies markedly with dataset properties. In particular, MUSIC-AVQA provides a lower-shortcut contrast where real training audio adds +6.69 pp beyond Silent training.

We also replicated the Normal and Silent question-level analysis on Qwen2-Audio-7B-Instruct and MiDashengLM-7B, observing question-level overlaps from 0.77 to 0.87 across benchmarks (with the parser sensitivity of one Qwen2-Audio MMSU row documented separately). Together, these experiments establish our Silent Twin measurement across three architectures, while full-pool STS editing is evaluated on Qwen2.5-Omni-7B.

[W2, W3, Q2] Fixed-coefficient transfer and Scaling

We applied the historical MMAR-selected pair (β,λ)=(0.4,0.3)(\beta, \lambda) = (0.4, 0.3) unchanged to other benchmarks and to the new training pools. This protocol tests fixed-coefficient transfer without target-specific retuning, and we treat MMAR strictly as the development set.

Training PoolMetricMMSUAIR-BenchFull MMAU-Pro
AVQASTS minus Normal (pp)+0.73+0.87+1.43
HeteroAQA-40KSTS minus Normal (pp)+0.96+1.48-0.84

Fixed STS reduces negative transfer across multiple tasks, while giving back part of the gain when Normal fine-tuning already transfers well, as on HeteroAQA-40K to MMAU-Pro.

Comparing STS against uniform Scaling matched to the same source-domain retention shows statistically positive STS gains on AIR-Bench for both AVQA (+0.20 pp) and HeteroAQA-40K (+0.14 pp), with smaller differences on the remaining benchmarks. These results separate the broad benefit of update-size control from the targeted contribution of Silent subtraction.

[Q3] Statistical significance

We quantify uncertainty using paired recording-cluster bootstrap confidence intervals and per-seed evaluations. For AVQA-trained STS (0.4, 0.3) versus same-β\beta Scaling:

BenchmarkSTS minus Same-β\beta Scaling95% CI
MMAU-mini+0.51 pp[0.10, 0.95]
MMSU+0.30 pp[0.02, 0.59]
AIR-Bench-0.02 pp[-0.13, 0.10]

The improvements on MMAU-mini and MMSU are statistically positive, while AIR-Bench is level at equal β\beta. At matched source retention, differences become smaller, and resampling benchmark-specific grid selection yields comparable performance between tuned STS and tuned Scaling. Our conclusions therefore distinguish reliable gains over Base and Normal from the smaller additional margin over matched Scaling.

Reviewer_zLpnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates what audio-language models learn when the fine-tuning objective is loosely defined. It introduces a "silent twin" baseline that replaces the training waveforms with equal-length silences, however, keeps all other training details the same. Both models are then tested on real audio. Training on silence recovers 79% of the AVQA improvements, and the gains/losses in several questions can be seen in the three different architectures. It proposes a new method called Silent-Twin Subtraction that uses the subtraction of the silent update and scales the overall updates. Experiments are carried out to look at the transfer, coefficient choice and preservation of original fine-tuning gains.

Critical strengths

The silent twin provides a well-controlled baseline. By preserving the lengths and token positions in the audio, it provides a clearer baseline than simply removing the audio input. Both models are evaluated on real recordings, and there is also a text-only baseline and a corpus audit. This gives strong evidence to show that there is a significant improvement when training in silence.

The question-level analysis is the best part of the paper. Whilst prior work (e.g. Omni-R1) has shown that audio-only training can improve results on audio benchmarks, this paper shows that many of the same changes happen when training with and without audio. The noise correction, separate Base panel and three different architectures mean that this observation is given much more weight than simply comparing overall accuracy.

Critical weaknesses

[W1] There is some exploratory evidence within the results provided. In Table 5 we can see the configurations chosen directly for each test set. While the grouped cross-validation is helpful, the partition was ultimately fixed after examining the full grids (Appendix D.3). This means it is not clear to what extent the method and evaluation were informed by the results. On AIR-Bench where the grid comparison was explicitly stated ahead of evaluation, STS has a slight advantage over scaling (71.83% vs 71.87%). This means there is clearer evidence that the model was tuned for each benchmark than independent validation of a transfer advantage. [W2] The other weight editing baselines show fairly limited tuning. There were 166 configurations tested for STS (Appendix F) and only 6 for LiNeS, Ext-Sub and DARE. Ext-Sub in particular suffers from having only 3 values for the global scale (0.5, 0.75 and 1). All the optimal values are at the lower limit. STS also prefers smaller values for both MMAR and MMAU-mini. While there is a reasonable comparison between STS and uniform scaling there is less convincing evidence that STS is better than the other methods due to the limited search space. [W3] In some cases there is a large impact on the original adaptation when increasing the transfer accuracy. On MMAU-mini STS has a 0.53 point advantage over scaling from cross-validation but does not retain any of the AVQA improvement whereas scaling retains 28%. This also happens with the fixed-point decomposition shown in Appendix E. Adding the orthogonal residual mainly helps the benchmark that the fixed point was chosen for. The AVQA improvement is retained less (81.4% to 23.6%). MMAR is promising as the retention is almost the same. However, for the rest of the tasks there is less evidence that the full subtraction helps over a simple rescaling correction.

Questions for authors

[Q1] Which choices were made in exploring the full benchmark grids? Are these only the fold allocations or do they also include the range of coefficients, comparator settings and other choices? [Q2] Could the observed ordering between Ext-Sub and LiNeS still be seen using larger ranges of tuning parameters? Ext-Sub’s optimal tuning was at the lowest possible scale so it is particularly important to consider extending the tuning parameters below 0.5. [Q3] At similar AVQA gains, how does STS compare to uniform scaling and the corresponding module-wise rescaling component? Does the orthogonal residual contribute at the operating points chosen by cross-validation? This question has only been considered at the single operating point in Appendix E.

1 comment
AuthorsOpenReview ↗

Response to Reviewer zLpn

[W1, Q1] Coefficient selection and evaluation history

We clearly separate retrospective grid analyses from fixed-coefficient transfer evaluations.

STS was developed with MMAR feedback, and we used the 166-cell grid and grouped five-fold CV for exploratory analysis. Therefore, we treat MMAR strictly as a development benchmark, and we classify the grid ranges and initial comparator settings as exploratory choices.

Our first frozen point, F=(0.5,1)F=(0.5, 1), was selected on MMAU-mini prior to transfer testing. On AIR-Bench, FF exceeded Base by +1.45 pp and trailed same-step Scaling by 0.40 pp. On the MMAU-Pro multiple-choice subset, FF trailed same-step Scaling by 0.98 pp. We report all of these preregistered results directly.

The AIR-Bench subgrid comparison was also registered before evaluation. Its best STS and Scaling scores reached 71.90% and 71.87%, giving a difference of +0.03 pp.

To test fixed-coefficient transfer, we evaluated the historical (0.4, 0.3) pair without retuning on the remaining benchmarks and on two independent training pools, MUSIC-AVQA and HeteroAQA-40K. Throughout our analysis, we keep these fixed-transfer results separate from benchmark-selected optima.

[W2, Q2] Expanded baseline tuning

We expanded LiNeS, Ext-Sub, and DARE from 6 configurations each to 14, 14, and 16 configurations, including Ext-Sub global scales below 0.5. All methods use the same grouped five-fold CV protocol.

MethodConfigsMMAR (%)MMAU-mini (%)MMSU (%)
Scaling1662.8474.5964.50
LiNeS1462.8875.0364.70
Ext-Sub1463.6374.9065.10
DARE1663.1774.7364.53
STS16663.8075.1264.83

Extending scales below 0.5 improves Ext-Sub by +0.93 pp on MMAR and +1.90 pp on MMAU-mini. Under this expanded search, STS remains competitive with the strongest tuned baseline, with differences of +0.17, +0.09, and -0.27 pp across the three benchmarks. All of these differences lie within statistical uncertainty. STS therefore matches strong weight-editing methods under fair tuning while grounding its reference update in a controlled Silent Twin run.

[W3, Q3] Source retention and directional contribution

Subtracting the Silent update changes both update size and update direction. We separate these two factors using uniform Scaling matched to the same source-domain gain. On AVQA, fixed STS (0.4, 0.3) retains 72% of Normal's in-domain gain, and Retention-matched Scaling retains 71%.

Training PoolMetricMMSUAIR-BenchMMAU-Pro
AVQASTS minus Matched Scaling (pp)+0.30+0.20-0.03
HeteroAQA-40KSTS minus Matched Scaling (pp)+0.05+0.14+0.15

The AIR-Bench gains over Matched Scaling are statistically positive on both training pools.

At (0.4, 0.3), module-wise rescaling alone differs from uniform Scaling by at most 0.07 pp across four benchmarks. We next replace the Silent residual with norm-matched random and other-seed Normal directions calibrated to the same source accuracy. STS exceeds both direction controls on AIR-Bench by +0.16 and +0.10 pp, and it exceeds the random direction on MMSU by +0.31 pp. On 1,000 MMSU questions, STS also increases the real-audio answer margin by 0.037 nats relative to Matched Scaling. This confirms a modest functional shift in option scoring while accuracy under audio interventions remains matched.

Overall, update shrinkage explains a large share of STS's transfer benefit, while the Silent direction adds consistent gains on specific tasks. Because all MMAR CV folds select (0.4, 0.3), our exact decomposition covers that CV operating point directly, while the remaining benchmarks select varying CV points across folds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.