Preserving Attention Functions for Replay-Free Continual Speech Recognition

Continual adaptation aims to learn from evolving data distributions while retaining previously acquired knowledge. In automatic speech recognition (ASR), variations in noise, recording conditions, and speaking styles make it difficult to adapt to new domains without degrading recognition on earlier ones, particularly when past utterances and transcripts are unavailable. Constraining updates on representations shared across speech domains can also restrict useful adaptation. We propose preserving attention functions (PAF), a method for replay-free continual ASR that regularizes composite attention operators during low-rank adaptation (LoRA). After each task, PAF summarizes attention inputs using their means and principal component bases. On the resulting affine supports, it jointly regularizes a score operator formed by query and key projections for context selection and a transport operator formed by value and output projections for information transformation. This objective permits coordinated changes in individual projections that leave the composite operators unchanged on the stored supports. PAF stores score references directly and reconstructs transport references from the merged model and a stored history of low-rank updates to the value and output projections. Preservation is evaluated without past examples or a separate teacher model. The learned updates are merged into a single ASR model, requiring no task identifiers or additional modules at inference. Experiments across multiple domain sequences and ASR models show lower final average word error rates than the reported replay-free baselines and reduced forgetting relative to sequential LoRA adaptation.