- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper investigates the fine-tuning dynamics of speech language models. It focuses on early recognition collapse, subsequent recovery, and long-term source-domain degradation. The authors fine-tune three speech language models on five Chinese dialect data with different learning rates. They compare loss and CER trajectories to analyze changes in recognition performance during fine-tuning. They further use bidirectional layer patching to examine cross-checkpoint functional incompatibility, and gradient similarity and spectral analysis to study optimization dynamics. Finally, they show that persistent source-domain degradation can still occur even when early collapse is avoided through the AdamW update transfer experiment.
Critical strengths
Systematic analysis of SFT dynamics.
The paper provides some interesting observations in investigating SFT dynamics across multiple speech language models and learning rates through practical and reasonable interventions.
Critical weaknesses
W.1 Unclear research motivation and problem formulation.
The Introduction does not clearly establish the research gap or define the specific problems the paper aims to investigate. In particular, the term "inference-time functionality" is vague for me, and the distinction between the supervised training objective and actual inference behavior is not sufficiently explained. Moreover, although some related studies are cited, their limitations are not clearly connected to the proposed analyses.
W.2 Limited novelty and depth of analytical insights.
The novelty of this work remains unclear for me. Prior studies [1, 2] have already investigated similar collapse-and-recovery phenomena in continual learning and domain adaptation where possible underlying causes are also analized. But these closely related works are not discussed. While this paper extends the analysis to speech language model fine-tuning, it is unclear what new understanding is gained beyond the existing findings. Besides, most analyses remain descriptive rather than providing deeper explanations of the observed phenomena.
[1] De Lange et al. Continual Evaluation for Lifelong Learning: Identifying the Stability Gap. ICLR 2023.
[2] Guo et al. Efficient Domain Continual Pretraining by Mitigating the Stability Gap. ACL 2025.
Questions for authors
1. Practical relevance of early collapse.
Would the observed collapse-and-recovery phenomena still be significant under more typical or carefully tuned warm-up settings used in practical SFT?
2. Mechanism behind recovery.
What actually drives the observed recognition recovery? Does the model progressively integrate new dialect knowledge with its original capabilities, or does continued target-domain learning improve Mandarin recognition through some shared mappings without necessarily restoring the original knowledge?
3. Definition of functionality.
Could the authors clarify what exactly is meant by "functionality" and "functional incompatibility" in this work?
4. Interpretation of bidirectional layer patching.
Cross-checkpoint incompatibility may also arise from ordinary representation changes during SFT. Could the author clarify whether the observed incompatibility is specifically associated with collapse in model recognition capability rather than general representation changes across checkpoints?
5. Choice of fine-tuning components.
Could the author clarify the reason that freeze the audio encoder and projector while only fine-tuning the LLM, where updating audio encoder might also be important in handling dialect adaptation?
Response to Reviewer HuXP
We thank the reviewer for recognizing the value of trajectory-level evaluation and causal intervention. We would like to clarify the central contributions of our work and address several concerns.
1. Loss and CER capture different aspects of training dynamics
We agree that evaluation loss can exhibit pronounced spikes, particularly at extremely high learning rates. Our claim is not that loss is insensitive to all forms of collapse, but that it does not fully characterize the severity and temporal structure of functional degradation and recovery.
As shown in Figs. 2 and 3, loss distinguishes broad optimization regimes, while CER reveals substantially different functional trajectories, particularly at intermediate learning rates.
The distinction between teacher-forced loss and autoregressive decoding is well known. Our contribution is to investigate how this discrepancy evolves throughout SFT and to connect observable functional changes with cross-checkpoint incompatibility and gradient dynamics.
2. AdamW's early-step behavior does not sufficiently explain the observed dynamics
The reviewer attributes early collapse primarily to AdamW's initial updates. While we acknowledge this potential contribution, the first-step update formula describes parameter displacement without determining its functional consequences.
Importantly, Appendix F (Fig. 10) already directly investigates the effect of learning-rate warm-up. Using the same AdamW optimizer and a 2% linear warm-up:
Furthermore, all three configurations reach similar final CER with and without warm-up.
These results show that reducing early update magnitudes does not consistently prevent functional degradation. Moreover, under constant learning rates, models can recover from severe collapse despite continued optimization at the same learning rate.
Therefore, AdamW's early-step behavior alone cannot account for the model-dependent collapse, recovery, and subsequent drift.
The central question of our work concerns how parameter updates translate into functional changes, and why these consequences vary across model states and optimization trajectories.
3. Collapse, repair, and drift are distinguishable functional dynamics
These terms characterize observable changes in recognition performance:
Our investigation goes beyond describing these trajectories.
Bidirectional layer patching reveals structured functional incompatibility that reorganizes during recovery. Gradient analysis characterizes the accompanying changes in optimization structure.
Most importantly, Fig. 7 provides intervention-based evidence that early collapse is not necessary for subsequent drift. By modifying the early optimization trajectory, we bypass pronounced collapse while substantial Mandarin degradation still emerges during continued training.
This separation is a central contribution that cannot be explained solely by AdamW's initial-update behavior.
4. Scope and practical significance
We deliberately investigate decoder adaptation under controlled conditions across three independently developed speech language models and three learning rates.
We do not claim that all SFT configurations exhibit identical dynamics or propose a universal mitigation method. The purpose is to identify and characterize functional and optimization phenomena that conventional loss monitoring and endpoint evaluation cannot fully reveal.
The practical relevance lies in understanding why apparently similar optimization conditions can produce substantially different functional trajectories, and why avoiding early collapse does not guarantee long-term capability retention.
In summary, our contribution is not the rediscovery of early optimization instability. It is the systematic characterization of functional collapse, recovery, and drift, supported by internal functional diagnostics, gradient analysis, and a causal intervention demonstrating that early collapse is not a prerequisite for later degradation.