acceptodds
Under review as a conference paper at ICLR 2027

Collapse, Repair, and Drift: Dissecting Fine Tuning Dynamics in Speech Language Models

Abstract

Supervised fine tuning (SFT) is commonly monitored through training and evaluation loss, but loss does not fully characterize how model functionality evolves along the same trajectory. We study this gap through dialect adaptation of Qwen3-ASR, MiMo-Audio, and FireRed-LLM across learning rates from 10⁻⁵ to 10⁻³. Character error rate (CER) reveals pronounced transient collapse and recovery whose severity and temporal structure are only weakly reflected by loss. Bidirectional layer patching reveals strong cross checkpoint functional incompatibility during recoverable collapse, with model dependent depth profiles that reorganize during recovery. Pairwise gradient similarity alone does not reliably indicate optimization stability, whereas raw gradient spectral analysis provides a clearer structural distinction between stable and unstable optimization: successfully converged runs approach a broad and temporally stable spectral energy distribution, while failed runs remain spectrally concentrated or exhibit persistent fluctuations. Finally, AdamW update transfer bypasses pronounced early collapse, yet substantial Mandarin degradation still emerges later in training. These results motivate three separable SFT dynamics: collapse, repair, and drift, and show that transient collapse is not necessary for persistent source domain degradation.

Primary area:
interpretability and explainable AI
Keywords:
Speech Language Models, Supervised Fine-Tuning, Training Dynamics, Gradient Geometry

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_npLhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates the fine-tuning dynamics of speech language models. It focuses on early recognition collapse, subsequent recovery, and long-term source-domain degradation. The authors fine-tune three speech language models on five Chinese dialect data with different learning rates. They compare loss and CER trajectories to analyze changes in recognition performance during fine-tuning. They further use bidirectional layer patching to examine cross-checkpoint functional incompatibility, and gradient similarity and spectral analysis to study optimization dynamics. Finally, they show that persistent source-domain degradation can still occur even when early collapse is avoided through the AdamW update transfer experiment.

Critical strengths

Systematic analysis of SFT dynamics.

The paper provides some interesting observations in investigating SFT dynamics across multiple speech language models and learning rates through practical and reasonable interventions.

Critical weaknesses

W.1 Unclear research motivation and problem formulation.

The Introduction does not clearly establish the research gap or define the specific problems the paper aims to investigate. In particular, the term "inference-time functionality" is vague for me, and the distinction between the supervised training objective and actual inference behavior is not sufficiently explained. Moreover, although some related studies are cited, their limitations are not clearly connected to the proposed analyses.


W.2 Limited novelty and depth of analytical insights.

The novelty of this work remains unclear for me. Prior studies [1, 2] have already investigated similar collapse-and-recovery phenomena in continual learning and domain adaptation where possible underlying causes are also analized. But these closely related works are not discussed. While this paper extends the analysis to speech language model fine-tuning, it is unclear what new understanding is gained beyond the existing findings. Besides, most analyses remain descriptive rather than providing deeper explanations of the observed phenomena.

[1] De Lange et al. Continual Evaluation for Lifelong Learning: Identifying the Stability Gap. ICLR 2023.

[2] Guo et al. Efficient Domain Continual Pretraining by Mitigating the Stability Gap. ACL 2025.

Questions for authors

1. Practical relevance of early collapse.

Would the observed collapse-and-recovery phenomena still be significant under more typical or carefully tuned warm-up settings used in practical SFT?


2. Mechanism behind recovery.

What actually drives the observed recognition recovery? Does the model progressively integrate new dialect knowledge with its original capabilities, or does continued target-domain learning improve Mandarin recognition through some shared mappings without necessarily restoring the original knowledge?


3. Definition of functionality.

Could the authors clarify what exactly is meant by "functionality" and "functional incompatibility" in this work?


4. Interpretation of bidirectional layer patching.

Cross-checkpoint incompatibility may also arise from ordinary representation changes during SFT. Could the author clarify whether the observed incompatibility is specifically associated with collapse in model recognition capability rather than general representation changes across checkpoints?


5. Choice of fine-tuning components.

Could the author clarify the reason that freeze the audio encoder and projector while only fine-tuning the LLM, where updating audio encoder might also be important in handling dialect adaptation?

Reviewer_HuXPOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies trajectory-level dynamics of supervised fine-tuning (SFT) in three LLM-based ASR models (Qwen3-ASR-1.7B, MiMo-Audio-7B, FireRed-LLM) adapted to five Chinese dialects. Training uses learning rates from 10⁻⁵ to 10⁻³, constant LR, no warm-up, and a single epoch. The authors report that character error rate (CER) reveals early transient collapse and recovery that evaluation loss reflects only coarsely, and that at 10⁻³ (and for MiMo-Audio at 10⁻⁴) the collapse becomes persistent failure. Bidirectional layer patching between pretrained and early SFT checkpoints shows depth-dependent, model-specific functional incompatibility that weakens during recovery. On the gradient side, pairwise cosine similarity between dialect gradients is argued to be an unreliable stability indicator, whereas the effective dimensionality of the 5-row raw gradient matrix separates converged runs (broad, stable spectra) from failed ones. Finally, an “AdamW update transfer” intervention lets a student follow scaled teacher updates and copies the teacher’s optimizer state; it bypasses the early collapse, yet Mandarin CER still degrades later. From this the authors conclude that collapse, repair, and source-domain drift are separable processes.

Critical strengths
  1. Useful trajectory-level perspective. Dense per-step CER tracking on both target and source domains, across three independently developed models and three learning rates, gives a much richer picture than endpoint-only forgetting evaluations.
  2. ASR is a well-chosen testbed. Errors are measurable against a clear reference, which makes it easy to observe functional degeneration (e.g., the repetition loops in Appendix G) that teacher-forced loss does not show.
  3. An attempt at causal intervention. The update-transfer experiment tries to separate collapse from drift by intervention rather than correlation alone, which is the right direction for this kind of claim.
  4. Clear and transparent reporting. The appendix fully specifies batching, gradient extraction, patching, and the intervention schedule. The warm-up ablation is included even though it does not neatly support a single story.
Critical weaknesses
  • The loss-vs-CER claim is overstated and not quantified. Figures 1 and 2 show that eval loss also spikes sharply during collapse, reaching about 200 for Qwen3-ASR at η=10−3\eta = 10^{-3}. The difference between the two views is mostly one of scale and visualization. CER values above 100% come from free-running repetition loops, a known gap between teacher-forced loss and autoregressive decoding. The paper reports no quantitative relationship between the two signals, such as rank correlation, lag, or detection delay.
  • Generality and practical value are limited. The study covers ASR only, Chinese dialects only, a private corpus, decoder-only fine-tuning with frozen encoders, one epoch, and a constant learning rate. This does not support broad claims about “SFT dynamics.” The paper also offers no actionable diagnostic or mitigation beyond common practice (warm-up, a lower η\eta, replay), and does not evaluate PEFT methods such as LoRA, where these dynamics may differ substantially.
  • The taxonomy is not operationalized. “Collapse,” “repair,” and “drift” are never defined quantitatively.
  • The collapse is largely an artifact of the setup, and the paper does not engage with known explanations. With bias correction, Adam’s first update is approximately$\Delta\theta_1 = -\eta \frac{g_1}{\sqrt{g_1^2} + \epsilon} \approx -\eta \operatorname{sign}(g_1).$ Every decoder parameter therefore moves by roughly η\eta in the first step, regardless of how small its gradient is. At η∈[10−4,10−3]\eta \in [10^{-4}, 10^{-3}] for full-decoder fine-tuning, early instability is expected. The paper does not connect the observed “collapse” to the literature on Adam’s early-step variance and the rationale for warm-up (e.g., RAdam and later analyses of why warm-up is needed), so the main phenomenon reads as a rediscovery of a known optimizer pathology.
Questions for authors

see above

1 comment
AuthorsOpenReview ↗

Response to Reviewer HuXP

We thank the reviewer for recognizing the value of trajectory-level evaluation and causal intervention. We would like to clarify the central contributions of our work and address several concerns.

1. Loss and CER capture different aspects of training dynamics

We agree that evaluation loss can exhibit pronounced spikes, particularly at extremely high learning rates. Our claim is not that loss is insensitive to all forms of collapse, but that it does not fully characterize the severity and temporal structure of functional degradation and recovery.

As shown in Figs. 2 and 3, loss distinguishes broad optimization regimes, while CER reveals substantially different functional trajectories, particularly at intermediate learning rates.

The distinction between teacher-forced loss and autoregressive decoding is well known. Our contribution is to investigate how this discrepancy evolves throughout SFT and to connect observable functional changes with cross-checkpoint incompatibility and gradient dynamics.

2. AdamW's early-step behavior does not sufficiently explain the observed dynamics

The reviewer attributes early collapse primarily to AdamW's initial updates. While we acknowledge this potential contribution, the first-step update formula describes parameter displacement without determining its functional consequences.

Importantly, Appendix F (Fig. 10) already directly investigates the effect of learning-rate warm-up. Using the same AdamW optimizer and a 2% linear warm-up:

  • FireRed-LLM at 10−410^{-4}: early collapse is substantially suppressed.
  • Qwen3-ASR at 10−410^{-4}: collapse is attenuated and delayed, but persists.
  • Qwen3-ASR at 10−510^{-5}: early Mandarin degradation becomes more pronounced.

Furthermore, all three configurations reach similar final CER with and without warm-up.

These results show that reducing early update magnitudes does not consistently prevent functional degradation. Moreover, under constant learning rates, models can recover from severe collapse despite continued optimization at the same learning rate.

Therefore, AdamW's early-step behavior alone cannot account for the model-dependent collapse, recovery, and subsequent drift.

The central question of our work concerns how parameter updates translate into functional changes, and why these consequences vary across model states and optimization trajectories.

3. Collapse, repair, and drift are distinguishable functional dynamics

These terms characterize observable changes in recognition performance:

  • Collapse: rapid deterioration of recognition functionality.
  • Repair: subsequent recovery from functional degradation.
  • Drift: longer-term source-domain degradation during continued adaptation.

Our investigation goes beyond describing these trajectories.

Bidirectional layer patching reveals structured functional incompatibility that reorganizes during recovery. Gradient analysis characterizes the accompanying changes in optimization structure.

Most importantly, Fig. 7 provides intervention-based evidence that early collapse is not necessary for subsequent drift. By modifying the early optimization trajectory, we bypass pronounced collapse while substantial Mandarin degradation still emerges during continued training.

This separation is a central contribution that cannot be explained solely by AdamW's initial-update behavior.

4. Scope and practical significance

We deliberately investigate decoder adaptation under controlled conditions across three independently developed speech language models and three learning rates.

We do not claim that all SFT configurations exhibit identical dynamics or propose a universal mitigation method. The purpose is to identify and characterize functional and optimization phenomena that conventional loss monitoring and endpoint evaluation cannot fully reveal.

The practical relevance lies in understanding why apparently similar optimization conditions can produce substantially different functional trajectories, and why avoiding early collapse does not guarantee long-term capability retention.

In summary, our contribution is not the rediscovery of early optimization instability. It is the systematic characterization of functional collapse, recovery, and drift, supported by internal functional diagnostics, gradient analysis, and a causal intervention demonstrating that early collapse is not a prerequisite for later degradation.

Reviewer_AbLrOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how speech recognition changes during dialect fine-tuning. It examines three speech language models across three learning rates and identifies early recognition collapse, subsequent recovery, and later degradation on Mandarin. It uses layer patching, gradient analysis, and AdamW update transfer to investigate these changes. A main finding is that avoiding pronounced early collapse does not necessarily prevent later Mandarin degradation.

Critical strengths
  1. The motivation is clear and practically useful. The comparison between loss and character error rate (CER) shows why monitoring loss alone may miss the severity of recognition failures during fine-tuning.
  2. Experiments across multiple models strengthen the observations. The study includes Qwen3-ASR, FireRed-LLM, and MiMo-Audio at three learning rates. The layer-patching and gradient analyses provide complementary views of their training dynamics.
  3. The intervention provides useful evidence beyond observing training curves. Figure 7 shows that later Mandarin degradation can occur even when pronounced early collapse is avoided. Appendix F also examines learning-rate warm-up, which helps address whether the observations depend on the training schedule.
Critical weaknesses
  1. The explanation of repair and drift is stronger than the evidence supports. The intervention supports the conclusion that early collapse is not necessary for later degradation. However, Section 7.2 further suggests that a stable Mandarin CER reflects ongoing repair offsetting ongoing forgetting. These two processes are not measured separately. Moreover, the intervention changes several factors together, including delayed updates, transferred optimizer states, and update magnitudes. Additional controls would help establish which factors produce the observed behavior.
  2. The additional value of the gradient spectrum needs further validation. The spectrum is computed from five dialect-specific batch gradients and reflects both their directions and magnitudes, as the authors acknowledge. Therefore, a more balanced spectrum may partly reflect more balanced gradient norms. Comparisons with normalized gradients and gradient-norm statistics would help establish what the spectral analysis adds beyond the existing similarity measurements. Its effective dimensionality is also limited to these five gradients, rather than describing the full parameter space.
  3. The repeatability of the findings is not sufficiently established. Results across multiple random seeds are not reported for the main trajectory comparisons. The update-transfer experiments explicitly use seed 5230. Although three models provide useful coverage, the consistency of collapse timing, recovery, and intervention effects within each setting remains unclear. The shaded bands in Figure 5 describe variation across dialects or dialect pairs, rather than uncertainty across repeated training runs.
  4. The experimental setting limits generalization. All models are adapted on the same private five-dialect corpus, with the acoustic components frozen and training restricted to one pass through the data. The warm-up experiment is useful, but examines only a 2% warm-up ratio in three configurations. A public dataset and additional training schedules would strengthen the broader conclusions about speech-model fine-tuning.
  5. The decoding limit may affect the observed collapse. Appendix A.3 sets the maximum generation length to twice the reference transcript’s character length. Since early failures include repetitive outputs, this limit may affect insertion errors and the apparent severity of collapse. Results using a fixed or audio-duration-based limit would help establish robustness.
Questions for authors
  1. What evidence distinguishes a plateau caused by repair offsetting forgetting from a model simply reaching a stable but degraded solution?
  2. Does the update-transfer result hold with simpler controls, such as delayed training, optimizer-state transfer alone, or a comparable update-scaling schedule?
  3. Does the raw-gradient spectrum remain more informative after controlling for gradient norms and batch variation?
  4. Do the main observations replicate across seeds and on a public dataset?
  5. How often does decoding reach the generation limit, particularly during collapse?
  6. Can the authors clarify the CER discrepancy in Table 3 and the mean-versus-median labeling in Figure 5 described below?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.