acceptodds
Under review as a conference paper at ICLR 2027

Preserving Attention Functions for Replay-Free Continual Speech Recognition

Abstract

Continual adaptation aims to learn from evolving data distributions while retaining previously acquired knowledge. In automatic speech recognition (ASR), variations in noise, recording conditions, and speaking styles make it difficult to adapt to new domains without degrading recognition on earlier ones, particularly when past utterances and transcripts are unavailable. Constraining updates on representations shared across speech domains can also restrict useful adaptation. We propose preserving attention functions (PAF), a method for replay-free continual ASR that regularizes composite attention operators during low-rank adaptation (LoRA). After each task, PAF summarizes attention inputs using their means and principal component bases. On the resulting affine supports, it jointly regularizes a score operator formed by query and key projections for context selection and a transport operator formed by value and output projections for information transformation. This objective permits coordinated changes in individual projections that leave the composite operators unchanged on the stored supports. PAF stores score references directly and reconstructs transport references from the merged model and a stored history of low-rank updates to the value and output projections. Preservation is evaluated without past examples or a separate teacher model. The learned updates are merged into a single ASR model, requiring no task identifiers or additional modules at inference. Experiments across multiple domain sequences and ASR models show lower final average word error rates than the reported replay-free baselines and reduced forgetting relative to sequential LoRA adaptation.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
continual learning, automatic speech recognition, low-rank adaptation, catastrophic forgetting, attention preservation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Gqp9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies replay-free continual automatic speech recognition, in which one ASR model is adapted to a sequence of acoustic domains using only current-domain data, with no stored past utterances, no task identifier, and a single merged model at inference. Sequential LoRA adaptation is parameter efficient, but each new update overwrites earlier domains, and existing continual-LoRA methods such as O-LoRA, InfLoRA, C-LoRA, and GainLoRA constrain the parameter-update space through orthogonality, subspace isolation, or gating. The authors instead propose Preserving Attention Functions (PAF), which preserves the function that attention performs rather than the parameters themselves. Each attention head is decomposed into a QK operator for context selection and a VO operator for information transport. After every domain, PAF builds a compact summary of the head's query and key/value inputs using PCA bases, means, and radii, which provides fixed coordinates that are independent of utterance length. During later adaptation it regularizes the drift of both composite operators on these supports, with loss weights derived from a bound on the attention-output change. Score references are stored directly, while transport references are reconstructed from the merged weights and a history of low-rank V/O updates, so the method needs neither replay nor a separate teacher model. All updates merge into a single model with about 15.48 MiB of additional storage and unchanged inference speed. Experiments on Whisper-small, OWSM, and S2T-Large across several domain orders and against six baselines show lower final and average WER and reduced forgetting, supported by component ablations and diagnostics on PCA versus random supports, shared cross-domain input directions, and storage and runtime cost.

Critical strengths

1.The paper makes a well-motivated move from constraining the LoRA update space to preserving the attention function itself. It regularizes the QK and VO composite operators on fixed, utterance-length-independent PCA supports rather than freezing the four projections separately, which permits coordinated internal changes while keeping the net function on old domains. This gives replay-free and teacher-free retention that merges into a single model with about 15.48 MiB overhead and unchanged inference speed (RTF 0.389).

2.The validation is thorough. It spans three ASR backbones, several domain orders, three seeds, component and efficiency analyses, and the unfavorable S2T result, where average WER is worse than GainLoRA, is reported rather than concealed.

Critical weaknesses

1.The theoretical guarantee does not hold under the paper's own experimental setting, yet the framing suggests a provable preservation result. Propositions 1 and 2 require zero preservation error and exact coverage of every past input within the bounded supports. With m = 16 the supports explain only about 46% of the energy, some inputs lie outside the radii, and the layer-wise constants in the full-model bound are never quantified. The method is therefore in practice a theory-motivated empirical regularizer, and the bound should not be read as a WER guarantee.

2.The advantage over the strongest baselines is not shown to be robust. On several orders the gain over the best baseline is only 0.6 to 0.85 WER points, and the main table reports standard deviations without a paired significance test. The closest replay-free attention baseline, Attention-Spectrum, was adapted to ASR by the authors rather than tuned by its original authors, which weakens the claimed superiority over attention-level retention, while the core "shared directions do not impede new learning" argument rests mainly on correlational evidence and the 45% NLL result itself reveals an unresolved tension on those directions.

Questions for authors

1.Propositions 1–2 assume zero operator error and exact coverage of every past input within the support radii, yet at m = 16 only about 46% of the energy is explained and some inputs lie outside the radii. Please report (i) the fraction of test inputs/positions that violate the radius bounds, (ii) the numerical value of the Prop. 1–2 bound on the real 36-block network and how the layer-wise constants accumulate, and (iii) whether any probabilistic or expected guarantee survives when coverage is incomplete. If not, please state explicitly that the theory motivates rather than guarantees preservation.

2.On several orders the margin over the best baseline is only 0.6–0.85 WER points. Can you provide a paired significance test (paired utterance bootstrap or a paired test across seeds) for PAF versus the best baseline in each order, with confidence intervals, so the margins can be judged against seed and utterance noise?

3.Removing Q/K/V updates along shared directions reduces the new-domain NLL gain by 45%, which also indicates that these directions matter for learning the new domain. How much of each current update lies inside versus outside the retained supports, and how does the new-domain NLL gain change as λ increases?

4.How were m = 16 and λ = 1.0 chosen, and are they fixed across all orders and backbones without access to later-domain test data?

5.Equation (46) recovers past V/O by subtracting accumulated low-rank updates. Appendix A.1 mentions FP64 over 36 blocks. How large is the error between the reconstructed T_s and the T_s computed directly from the original checkpoint, in FP64 and in the FP16/BF16 precision a deployment would use, and does this error accumulate over six domains?

4 comments

Thank you for your detailed feedback on our work.

W1. Conditions for the theoretical guarantees and scope of the empirical results

As explained in Section 3.3, the conditional bound on changes in the attention function motivates the design of the PAF loss as a surrogate objective and the choice of its coefficients. The gains in recognition accuracy and reductions in forgetting are empirical, and we assessed the practical value of the objective through component comparisons and a controlled intervention.

Proposition 1 bounds the squared norm of the change in a head’s attention output by its preservation loss when the current and reference heads receive the same inputs on the bounded stored support and use the same allowed keys and additive mask.

∥fθ,s,h−fs,h∥22≤ℓs,h(θ).\lVert f_{\theta,s,h}-f_{s,h}\rVert_2^2 \leq \ell_{s,h}(\theta).

This inequality also holds for positive preservation loss. Proposition 2 extends the analysis to the complete model when the current and reference models differ only in the regularized attention projections and use the same acoustic input, decoder prefix, and attention masks. Under finite sensitivity bounds, Equations (42)–(44) bound the change in logits and the increase in mean token NLL, and Equation (43) accounts for incomplete coverage through the support residual ρs,ℓ(a,p)\rho_{s,\ell}(a,p).

Zero preservation error and complete coverage are sufficient conditions for exact recognition preservation, as stated after the proof of Proposition 2. Under these conditions, the same deterministic decoding procedure and tie breaking rule preserve the transcription and its WER. We therefore distinguish the conditional bounds in Propositions 1 and 2 from the sufficient conditions for exact recognition preservation.

The experiments also show the effect of PAF under limited input coverage. At the same m=16m=16, PAF reduced forgetting relative to TT only from 1.55 to 0.90 percentage points in Order 2 and from 6.32 to 3.67 percentage points in Order 3. PAF also had lower Last WER, Avg WER, and forgetting than Individual in both orders (Table 4). We clarified the relationship between the theoretical conditions and the observed performance gains in Section 3.3 and Appendix C.2.

Q1. Radius violations, the numerical bound, and the guarantee under incomplete coverage

  1. Radius violations. At m=16m=16, approximately 0.03% of queries and 0.01% of keys had augmented PCA coordinates outside the stored radii. These percentages measure whether the projected coordinates lie within the stored radii. For the original inputs, the retained subspaces explained 43.38% of the centered input energy for queries and 45.98% for keys (Appendix B.2).
  2. Numerical bound for the 36-block network. Equation (43) sums the per head error bounds within each block, adds the support residual term, and accumulates the block contributions weighted by their downstream sensitivities κℓ\kappa_\ell.
  3. Guarantee under incomplete coverage. The deterministic bound in Proposition 2 holds for each acoustic input and decoder prefix. The paper gives no probabilistic guarantee or upper bound on expected WER. Zero preservation error and complete coverage are sufficient for exact recognition preservation under the same deterministic decoder. In the experimental setting, the theoretical claim is limited to the role of the PAF loss as a surrogate objective (Appendix C.2).

W2. Robustness of the performance gains, comparison with Attention-Spectrum, and evidence for adaptation in shared directions

We interpret PAF’s gains within the scope of the backbones, domain orders, and metrics evaluated. Table 2 reports means and standard deviations over three seeds, and PAF had the lowest mean Last WER and Avg WER in all four Whisper-small orders. We also used a paired utterance bootstrap in Order 3 to evaluate the difference in Avg WER between PAF and O-LoRA. Q2 below gives the procedure and scope of this test.

We evaluated our ASR implementation of Attention-Spectrum. Section 4.3 specifies the regularization weight β=0.5\beta=0.5, and Table 19 in Appendix D describes the retained spectral and angular statistics and the preservation penalty. The comparison therefore applies to this ASR implementation and evaluation setting. We clarified the ASR adaptation and its evaluation setting in Section 4.3 and Appendix D so readers can assess the scope of the comparison.

The intervention in Table 8 shows that updates along shared input directions contributed to learning the new domain. Removing the Q/K/V update component along past input directions and matching the total update norm reduced the one step improvement in TED2 NLL by 45.13%. To examine these directions during adaptation, we compared λ=0,0.1,1\lambda=0,0.1,1 from the same checkpoint. Increasing λ\lambda from 0 to 1 reduced the share of update energy in the retained subspace from 15.36% to 4.12%, while the decrease in TED2 NLL grew from 1.0478 to 1.2254, and the mean NLL increase on the two past domains fell from 0.3145 to 0.1072. On this transition, PAF retained updates along shared directions while improving both retention and adaptation. Establishing the causal contribution of individual compensating Q/K components requires further analysis, and in Appendix A.3 we limited our interpretation of shared directions to the transition evaluated in Table 9.

Q2. Paired significance test against the best baseline

We used a paired utterance bootstrap to compare Avg WER between PAF and O-LoRA on Whisper-small in Order 3. O-LoRA had the lowest three seed mean Avg WER among the baselines in this order (Table 2). With the seed 2022 checkpoints fixed, PAF’s Avg WER was 0.645 percentage points lower than O-LoRA’s, with a 95% confidence interval for the improvement (O-LoRA minus PAF) of [0.448, 0.845] percentage points.

Within each domain, we sampled evaluation utterances with replacement, keeping both methods’ decoding outputs from all three adaptation stages together. This preserved the pairing across methods and stages. In each bootstrap replicate, we recomputed corpus WER for all nine combinations of domain and stage and used Equation (24) to calculate the difference in Avg WER. The entire confidence interval lies above zero, indicating significantly lower Avg WER for PAF when accounting for variation across evaluation utterances with the two checkpoints fixed.

The 0.6–0.85 percentage points mentioned by the reviewer refer to improvements in Last WER across several orders. Our additional test covers Avg WER in Order 3 for the seed 2022 checkpoints, and tests covering Last WER differences in each order and variability across training seeds remain open. Section 5.1 now mentions this test, and Appendix A.5 reports the bootstrap procedure, confidence interval, and scope of the test.

Q3. Share of update energy inside and outside the retained support and the effect of the preservation weight

Updates along past input directions contributed to adaptation on TED2. Removing the Q/K/V update component along those directions and rescaling the remaining update to the original total norm reduced the one-step improvement in TED2 NLL by 45.13% (Appendix A.3, Table 8). PAF allows learning along these directions by penalizing changes in the QK and VO operators on past inputs.

To measure the share of updates along shared directions during training, we adapted to TED2 from the same checkpoint after LS and CHiME3, varying only λ\lambda. Following Equations (36)–(37), we decomposed the AdamW updates to the Q/K/V weights into a component in the span of the PCA bases and input means of the two past domains and a component in its orthogonal complement. We computed the energy shares below by summing the squared Frobenius norms of each component over 142 updates sampled at intervals of 100 steps. We measured the decrease in TED2 NLL from the same starting value of 1.5917 in all three settings.

λ\lambdaEnergy in past subspaceEnergy in orthogonal complementFinal TED2 NLL (↓)Decrease in TED2 NLL (↑)Mean NLL increase on the two past domains (↓)
015.36%84.64%0.54391.0478+0.3145
0.15.56%94.44%0.44931.1424+0.1107
14.12%95.88%0.36631.2254+0.1072

On this transition, increasing λ\lambda from 0 to 1 reduced the share of update energy along shared directions from 15.36% to 4.12% and yielded a larger reduction in TED2 NLL, from 1.0478 to 1.2254. The mean NLL increase on the two past domains also fell from 0.3145 to 0.1072. At the default λ=1\lambda=1, some update energy remained along shared directions, and the new domain showed the largest NLL decrease among the three settings. We added the measurement procedure and results to Appendix A.3 (Table 9).


Q4. Choice of PCA dimension and preservation weight and use of test data

λ\lambdaMean WER on the two past domainsWER on the new domain (TED2)Mean WER on all three domains
0.111.0310.6010.89
0.511.2610.5511.02
1.010.8010.2810.63

As described in Section 4.1 of the submitted paper, we split the adaptation data into training and development subsets in a 9:1 ratio and used the development subset for validation. The training, development, and test subsets were disjoint, and we used the same splits for all methods and seeds. We fixed m=16m=16 and λ=1.0\lambda=1.0 across all main experiments, without retuning either value for individual domain orders or backbones.

We selected mm and λ\lambda using development results and fixed them before the main evaluation, without using test results from later domains to select or retune either hyperparameter.

Table 15 reports the performance–storage trade-off for m=16m=16, and Table 13 reports sensitivity to the preservation weight. In an additional evaluation of the existing checkpoints on 1,015 development utterances, λ=1.0\lambda=1.0 again gave the lowest mean WER on the past domains and the lowest WER on the new domain. This evaluation provides further support for the default setting.

Q5. VO reference reconstruction error across precisions and error accumulation

The submitted paper evaluated one six domain training order. To test whether the VO references could be accurately reconstructed after repeated merges in another order, we ran an additional experiment on Whisper-small with a random ordering of the six domains. The training seed was 2024, and the order was Common Voice → GigaSpeech-YOU → CHiME3 → GigaSpeech-POD → TED2 → LibriSpeech-other.

  1. Reconstruction error by precision after repeated merges. We reconstructed the past VO references T^s\widehat T_s using Equations (46)–(47) from the checkpoint after merging the final domain’s updates. We compared each reconstructed reference with the FP64 reference TsT_s computed directly from the original checkpoint for that domain. The table reports the maxima over 180 comparisons across five past domains and 36 attention blocks. We computed the relative error as ∥T^s−Ts∥F/∥Ts∥F\lVert\widehat T_s-T_s\rVert_F/\lVert T_s\rVert_F.

    Reconstruction precisionMaximum relative Frobenius errorMaximum absolute elementwise error
    FP648.87×10−88.87\times10^{-8}4.37×10−84.37\times10^{-8}
    FP168.40×10−48.40\times10^{-4}1.26×10−31.26\times10^{-3}
    BF167.00×10−37.00\times10^{-3}5.35×10−35.35\times10^{-3}

    Reconstructing the reference for Common Voice, the first domain, requires subtracting the V/O updates of all five subsequent domains from the final merged weights. Including this longest update history, the maximum relative error was 0.084% in FP16 and 0.700% in BF16. The final relative reconstruction error thus remained below 1% after five subsequent merges.

  2. Recognition performance on the same domain order. We evaluated all methods on the same 11,678 utterances.

    MethodLast WER (↓)Avg WER (↓)Forgetting (↓)
    IncLoRA25.2321.9310.06
    O-LoRA20.8119.616.56
    GainLoRA19.6819.165.36
    PAF19.4918.535.13

    On this additional six domain order, PAF had the lowest Last WER, Avg WER across all adaptation stages, and mean forgetting among the four methods. Compared with GainLoRA, the strongest baseline, PAF reduced Last WER by 0.19 percentage points, Avg WER by 0.63 points, and mean forgetting by 0.23 points.

Reviewer_G7xTOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a replay-free continual adaptation method for ASR to reduce catastrophic forgetting without retaining data from previous domains. The authors first use PCA to construct low-dimensional subspaces from the attention input representations of previous domains. Then, they represent the composite QK and VO functions on these subspaces and use the differences from their previous references as regularization during LoRA adaptation to new domains. In this way, the method aims to preserve previously learned attention functions while adapting to new domains. The authors evaluate their method on three ASR models across five domain sequences. They compare with six baseline methods adapted to ASR, including IncLoRA, EWC, O-LoRA, C-LoRA, GainLoRA, and Attention-Spectrum. The results show improvements in final average WER over the evaluated baselines and reduced forgetting issues.

Critical strengths

1. Technically sound methodology with reasonable theoretical justification.

The paper is technically good with readable mathematical formulations and reasonable theoretical analyses to support the methodology design.


2. Practical solution for replay-free continual adaptation for ASR.

The methodology provides a practical and reasonable approach to mitigating catastrophic forgetting in continual adaptation for ASR without requiring access to previous actual data samples.

Critical weaknesses

Insufficiently defined research gap and related-work.

The paper discusses several LoRA-based adaptation methods but does not clearly establish their limitations, with the motivation primarily focusing on a limitation of GainLoRA. The attention function preservation and replay-free learning are disccused without sufficiently explaining why existing approaches are not effective or what specific gap comapred to existing work PAF aims to address. Moreover, the main text lacks a systematic discussion of related continual ASR studies which makes it difficult to understand how PAF differs from existing approaches in continous adaptation. Consequently, the motivation and methodological novelty of this work is unclear for me which make me difficult to assess the significance of its contributions.

Questions for authors

Q1. Choice of PCA dimension.

Why was m=16m=16 selected as the default?


Q2. Comparison with ASR-specific continual learning methods.

The experiments primarily compare PAF with general continual learning methods adapted to ASR. Could the authors clarify whether there are existing ASR-specific continual learning or adaptation approaches relevant to this setting? If so, why were they not included in the comparisons? If no suitable methods are available under a comparable replay-free setting, please clarify this.


Q3. Domain sequence selection

What motivated the selection of the five domain orders?


Q4. Applicability to other ASR architectures

Could the authors clarify whether PAF can be applied to other CTC-based ASR models, such as wav2vec2?


Q5. Missing non-adaptation baseline.

The experiments do not include the original pretrained ASR model without any adaptation. Given the strong cross-domain generalization of some pretrained ASR models, reporting the WER of the model without adaptation may support the gain from adaptation using their methods more sound.

4 comments

We appreciate your thoughtful review and constructive suggestions.

W1. Research gap, novelty, and related work on continual ASR

The Introduction starts from the question of which updates to allow when earlier and current domains share representations useful for recognition. We discuss how existing methods control interference through task subspaces, parameter overlap, or branch responses, and explain why suppressing new responses along shared directions can also limit adaptation to the current domain. GainLoRA illustrates this issue. The next paragraph explains how individual projections can change together while preserving the QK and VO operators on the retained supports.

In the same paragraph, we explain that SEEKR uses replayed inputs to evaluate past behavior and that Attention-Spectrum preserves spectral statistics of attention maps without replay. PAF asks how to measure the changes a new update makes to context selection and information transport on earlier domains without access to their utterances. PAF directly measures changes in the composite QK and VO operators on the supports retained from those domains.

We revised the last sentence of the GainLoRA paragraph to explain why update directions or branch responses alone do not directly measure the combined effect of changes to several projections. Immediately after the discussion of SEEKR and Attention-Spectrum, we added the research question above so that the following paragraph introduces PAF as an answer.

In the ASR-specific baselines paragraph of Appendix D, we compare existing continual ASR methods, which retain performance on earlier domains by adding language specific components or by constraining or merging low-rank updates, with PAF, which controls changes in composite attention operators on retained supports.

Q1. Choice of the default PCA dimension m=16m=16

mLast WER (↓)Forgetting (↓)Retained storage
817.535.059.43 MiB
1616.633.8115.48 MiB
6415.241.9165.08 MiB

As stated in Section 4.1 of the original submission, we split the adaptation data into training and development subsets in a 9:1 ratio and used the development subset for validation. All methods and seeds used the same disjoint training, development, and test subsets. We selected mm and λ\lambda on development data and fixed them before the main evaluation, without using test results from later domains to select or retune either hyperparameter.

We chose m=16m=16 to balance recognition performance and retained storage. In Table 15 (Appendix B.4), increasing mm from 8 to 16 increased storage from 9.43 to 15.48 MiB, reduced Last WER from 17.53% to 16.63%, and reduced forgetting from 5.05 to 3.81 percentage points. At m=64m=64, recognition improved further and storage reached 65.08 MiB. These results support m=16m=16 as a practical trade-off between recognition performance and storage.

Q2. Comparison with ASR-specific continual learning methods

As discussed in W1, PAF asks which updates to allow when earlier and new domains share input directions useful for recognition. We chose GainLoRA as a primary baseline in the original submission because it allows us to compare two criteria for protecting earlier inputs directly. GainLoRA suppresses the contribution of a new LoRA branch along past input directions. PAF allows updates along those same directions and penalizes them according to how much they change QK context selection and VO information transport on earlier inputs.

To test whether this distinction affects speech domain adaptation, we ran a controlled experiment starting from the same checkpoint and released only GainLoRA’s gate update constraint. Releasing the constraint reduced WER on the new domain, CHiME3, by 3.29 percentage points and increased mean WER on the two earlier domains by 0.05 points (Appendix A.3, Table 10). This directly shows that a constraint protecting past input directions can affect learning on a new speech domain. We prioritized GainLoRA in the original submission because this control isolates the effect of the constraint on shared directions.

Following the reviewer’s suggestion, we also evaluated CSSVD and Ma-RPGP to compare PAF with ASR-specific methods that do not replay past utterances. CSSVD protects the leading singular directions of the weights and adapts in the subspace associated with smaller singular values. Ma-RPGP distills the previous model’s predictions on current speech and reduces the component of the new domain gradient that conflicts with the distillation gradient.

MethodLast WER (↓)
CSSVD16.93
Ma-RPGP17.29
PAF16.63

PAF preserves QK and VO on past inputs. Its final mean WER was 16.63%, 0.30 pp below CSSVD and 0.66 pp below Ma-RPGP. After Table 19, Appendix D explains how both baselines retain knowledge and reports all three WERs in Table 20.

Q3. Choice of the five domain orders

We selected the five orders to assess how training order affects forgetting on the same set of speech domains, how performance changes as more domains must be retained, and how the model transfers between speaking environments (Table 1).

In Orders 1–3, noisy speech (CHiME3), read speech (LibriSpeech-other), and lecture speech (TED-LIUM 2) each appear first once. Order 4 adds Common Voice, GigaSpeech-YOU, and GigaSpeech-POD to those three domains. At the final stage, the model must retain five earlier domains with different recording conditions and speaking styles, allowing us to assess retention over a longer and more varied sequence. In Order 5, we evaluated S2T-Large on lecture speech → noisy speech → YouTube speech to test retention under a domain order absent from the preceding experiments (Table 3). Placing noisy speech immediately after lecture speech introduces a direct transition absent from the other orders.

Q4. Extension to CTC and transducer ASR architectures

We chose Whisper as our main model to evaluate PAF on a modern end-to-end architecture that supports multiple speech tasks. This study focuses on continual ASR, and we evaluated PAF on OWSM and S2T-Large in addition to Whisper.

Following the reviewer’s suggestion, we identified CTC and transducer models as targets for future evaluation of PAF in Appendix E of the revised manuscript.

Q5. Comparison with the pretrained baseline without adaptation

Pretrained Whisper-small achieved a mean WER of 18.20% across the three evaluation domains without further training. On the same domains, the final PAF model after sequential adaptation in Order 3 achieved a mean WER of 16.63%, 1.57 percentage points lower. In Appendix B.4, immediately below Table 15, we added the mean WER of unadapted Whisper-small and specified the 1.57 point reduction in final mean WER on the same three domains.

Thank you for the detailed responses and additional experiments. Most of my concerns have been addressed. I just have a few minor suggestions for the revision:

  1. Could you clarify which domain order was used for the results in Table 20 when comparing with ASR-specific continual learning methods?
  2. It would be helpful to briefly mention the additional ASR baselines and the unadapted Whisper-small comparison in the main text, with references to the corresponding appendix sections.
  3. Given the expanded 10-page limit, I suggest moving the limitations into the main text to make the current scope clearer, particularly regarding CTC and transducer ASR models.

Thanks again for the effort put into the revision! I will increase the score.

Thank you for taking the time to review our responses and consider the additional experiments. We are glad that our responses addressed most of your concerns, and we appreciate your decision to raise your score! We have incorporated all three of your suggestions into the manuscript.

  1. The revised caption of Table 20 now states, “All three methods used Order 3 (Whisper-small, training seed 2022).” Table 1 gives the domain sequence for Order 3 as LibriSpeech-other → CHiME3 → TED2.
  2. Section 5.1 now briefly reports the comparisons with CSSVD and Ma-RPGP (Appendix D, Table 20) and with unadapted Whisper-small (Appendix B.4).
  3. We moved the limitations from Appendix E to the “Limitations and future work” paragraph in Section 6. This paragraph identifies evaluation on CTC-based and transducer-based ASR models as future work.
Reviewer_NijAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Preserving Attention Functions (PAF), a replay-free continual ASR method that regularizes composite attention operators during sequential LoRA adaptation. By preserving query-key routing and value-output transformations on PCA-based representations of previous domains, PAF aims to reduce catastrophic forgetting while allowing coordinated parameter updates. Experiments across multiple ASR architectures and domain sequences demonstrate improved recognition performance and reduced forgetting compared to several continual-learning baselines.

Critical strengths
  • The paper introduces an interesting approach to continual learning by preserving composite attention functions rather than individual projection weights. The theoretical analysis provides a principled motivation for jointly regularizing context selection and information transport while allowing coordinated parameter updates.

  • The experimental evaluation demonstrates consistent improvements across multiple domain sequences and ASR architectures. The component ablations and controlled interventions further support the effectiveness of joint attention preservation compared to individual projection regularization.

Critical weaknesses
  • The theoretical guarantees are established under restrictive assumptions that the current and reference models receive identical attention inputs lying within the stored supports. However, sequential adaptation changes intermediate representations throughout the network, and the complete-model bound depends on unspecified sensitivity constants and support residuals. Consequently, the analysis does not establish a practically meaningful guarantee against forgetting under realistic representation drift.

  • The retained PCA subspaces capture only a limited fraction of the original representation variation. In Appendix B.2, the default dimension of 16 explains approximately 43–46% of the input energy, with mean relative residuals around 0.70. Although larger subspaces improve recognition and retention, their additional storage costs are substantial. This raises questions about whether the proposed regularizer adequately preserves past attention behavior outside the selected subspaces.

  • Some baseline comparisons involve different preservation objectives, regularization strengths, and support constructions. In particular, the Individual baseline uses unweighted projection-level penalties, whereas PAF employs radius-weighted composite penalties. Although Appendix A.4 examines several regularization strengths, the comparison does not fully isolate the advantages of composite preservation from differences in effective regularization. Moreover, PAF does not consistently achieve the lowest forgetting among all baselines.

  • The evaluation is restricted to English ASR adaptation sequences containing at most six domains. Since PAF retains domain-specific PCA summaries and a history of low-rank updates, storage and preservation computation increase as additional domains arrive. Without experiments involving substantially longer task sequences or fixed memory budgets, its scalability to realistic lifelong adaptation remains insufficiently demonstrated.

Questions for authors
  • Can the authors provide empirical measurements of the complete-model stability bound, including support residuals and representation drift across adaptation stages? How strongly do these quantities correlate with actual forgetting?

  • How does PAF perform when the PCA subspaces capture substantially less of the input variation, or when domain distributions overlap heavily outside the retained subspaces? Can the authors provide a clearer performance–storage trade-off across support dimensions?

  • Can the authors compare PAF against individually regularized projections under more closely matched preservation strengths and computational budgets? How do the methods compare along the retention–adaptation Pareto frontier?

  • How does the retained memory and computational cost scale over longer sequences, particularly under a fixed memory budget? Can the proposed framework maintain its advantages when old domain summaries must be compressed or discarded?

5 comments

Thank you for taking the time to read our manuscript so carefully.

W1. Assumptions underlying the theoretical guarantees and representation drift

As explained in Section 3.3, our theoretical analysis gives a conditional bound on changes in the attention function that motivates the design of the PAF loss and its coefficients. This bound provides a theoretical basis for using the PAF loss as a surrogate objective for mitigating forgetting. The gains in recognition accuracy and reductions in forgetting are empirical, and we assessed the practical value of the objective through component comparisons and a controlled intervention.

Equation (19) of Proposition 1 shows that when the current and reference heads receive the same inputs on the stored support for domain ss and use the same allowed keys and additive mask, the squared norm of the difference between their outputs is bounded by the preservation loss for head hh.

∥fθ,s,h−fs,h∥22≤ℓs,h(θ).\lVert f_{\theta,s,h}-f_{s,h}\rVert_2^2 \le \ell_{s,h}(\theta).

Equations (42)–(43) of Proposition 2 assume finite sensitivity bounds. For each attention block, we sum the per head deviation bounds, add the contribution of the support residual, and weight the result by the downstream sensitivity. For the same acoustic input aa and decoder prefix pp, the logit deviation satisfies

∥gθ(a,p)−gs(a,p)∥∞≤Bs(a,p)\lVert g_\theta(a,p)-g_s(a,p)\rVert_\infty \le B_s(a,p)

Bs(a,p)=∑ℓ∈Bκℓ[∑h=1Hℓℓs,ℓ,h(θ)+(Lθ,ℓ+Ls,ℓ)ρs,ℓ(a,p)].B_s(a,p)=\sum_{\ell\in\mathcal B}\kappa_\ell\left[\sum_{h=1}^{H_\ell}\sqrt{\ell_{s,\ell,h}(\theta)}+(L_{\theta,\ell}+L_{s,\ell})\rho_{s,\ell}(a,p)\right].

This bound also holds when input coverage is incomplete and ρs,ℓ(a,p)>0\rho_{s,\ell}(a,p)>0. By reducing each head’s preservation loss, PAF reduces the terms in the accumulated deviation bound that capture changes in QK and VO. Equation (44) relates this bound to the increase in mean token negative log-likelihood on past transcripts.

Q1. Measuring the complete-model bound and its relationship to forgetting

Equation (39) in Appendix B.2 reports the mean relative residual rˉ\bar r, obtained by projecting each nonzero centered input z=x−μz=x-\mu onto the PCA subspace, dividing the Euclidean norm of its residual by ∥z∥2\lVert z\rVert_2, and averaging over inputs.

At m=16m=16, rˉ\bar r was 0.7010 for queries and 0.6998 for keys. In Equation (43), ρs,ℓ(a,p)=∥Is,ℓ−I^s,ℓ∥\rho_{s,\ell}(a,p)=\lVert I_{s,\ell}-\hat I_{s,\ell}\rVert measures the distance between the reference query and key/value inputs to attention block ℓ\ell and their projections onto the bounded stored supports for a given acoustic input and decoder prefix. We take the maximum Euclidean distance over positions and input streams. The two measured relative residuals show that input coverage is incomplete.

Even with this incomplete coverage, jointly preserving QK and VO with PAF reduced mean forgetting relative to TT only at the same m=16m=16, from 1.55 to 0.90 percentage points in Order 2 and from 6.32 to 3.67 percentage points in Order 3 (Table 4). Figure 3 shows how operator drift on the stored LS PCA16 support relates to forgetting on LS in a single Order 3 run. All three variants started from the same LS checkpoint (9.9154% WER). When we preserved only QK, VO changed by 76.98% and final LS WER reached 18.74%. When we preserved only VO, QK changed by 30.62% and final LS WER reached 16.50%. Preserving both limited the changes in QK and VO to 0.001182% and 0.2168%, respectively, and yielded the smallest increase in LS WER, from 9.9154% to 14.48%.

For each transition, we used 64 development utterances to measure the support residual, representation drift, the upper bound on logit change, and the observed logit change, and we computed corpus ΔWER on the same utterances.

TransitionSupport residual (ρˉ\bar\rho)Representation drift (Dˉ\bar D)Mean upper bound (Bˉ\bar B)Mean observed logit change (Aˉ\bar A)Corpus ΔWER (pp)
LS, T1→T244.5847.6224.494.96+4.05
LS, T1→T344.5852.1134.267.76+5.49
CH3, T2→T344.8352.6022.835.93+3.74

For each transition, ρˉ\bar\rho and Dˉ\bar D denote the means of ρi\rho_i and DiD_i over 64 development utterances. For utterance ii, the support residual ρi\rho_i is the maximum distance from the reference model’s attention inputs to the supports bounded by the stored radii. Representation drift DiD_i is the maximum difference between the current and reference models’ attention inputs. Both maxima are taken over blocks and positions. We computed the mean upper bound Bˉ\bar B and mean observed logit change Aˉ\bar A over the same reference prefixes and utterances.

In table order, Spearman correlations with utterance-level ΔWER were 0.21, −0.05, and −0.17 for ρi\rho_i, and 0.17, −0.05, and 0.10 for DiD_i. The correlations between ΔNLL and ΔWER were 0.39, 0.44, and 0.64, with respective 95% confidence intervals of [0.10, 0.64], [0.17, 0.67], and [0.38, 0.84] from 10,000 utterance-level bootstrap resamples.

W2. Input coverage of the PCA support and preservation beyond it

At m=16m=16, the PCA supports explained 43.38% of the centered input energy for queries and 45.98% for keys (Figure 7). Despite this limited input coverage, PCA16 captured 54.77% of the weight gradient energy, compared with approximately 2.09% for Random16 at the same support dimension (Figure 5). At the same m=16m=16, jointly preserving score and transport with PAF reduced mean forgetting relative to TT only, from 1.55 to 0.90 percentage points in Order 2 and from 6.32 to 3.67 percentage points in Order 3 (Table 4).

In Figure 6, we evaluated the model trained with m=16m=16 on the common LS PCA64 evaluation support, which is broader than the PCA16 support used during training. The relative operator changes were 1.58% for QK and 18.42% for VO. Table 4 thus shows that PAF helps reduce forgetting even with limited input coverage.

Q2. Performance–storage trade-off under varying support coverage and domain overlap

We measured the performance–storage trade-off across support dimensions in a single seed sweep for Order 3 (Table 15). For m=4,16,64m=4,16,64, respectively, Last WER was 18.07%, 16.63%, and 15.24%, forgetting was 5.22, 3.81, and 1.91 percentage points, and retained storage was 6.64, 15.48, and 65.08 MiB. Increasing mm from the default of 16 to 64 reduced forgetting by 1.90 percentage points and required about 4.2 times as much storage.

We have limited evidence for settings in which the domain distributions overlap substantially outside the retained subspaces. When we evaluated the model trained with m=16m=16 on the broader common LS PCA64 evaluation support, the relative operator changes were 1.58% for QK and 18.42% for VO (Figure 6).

W3. Comparison with Individual and results on forgetting

Individual penalizes changes in Q, K, V, and O separately. PAF penalizes changes in two composite operators on the stored supports. QK determines attention scores, and VO determines how each value input contributes to the head output. In Order 3, PAF’s O weights changed by 22.23% on average across 432 heads, yet the mean change in VO on the stored LS support was only 0.2168% (Table 5). With Individual, VO changed by 36.87%. Mean QK drift was also smaller with PAF (0.001182%) than with Individual (1.59%), and final LS WER was 14.48% with PAF and 16.44% with Individual. To examine how this difference affects useful adaptation, we removed the component of the O update that preserves QK and VO on both past domains, starting from the same PAF checkpoint (Appendix A.2). The PAF preservation loss remained constant, Individual’s O penalty for LS decreased in all 36 blocks, and NLL on the new domain, TED2, increased from 0.329 to 0.571. This shows that Individual can penalize an update that contributes to prediction on the new domain while preserving the past domains’ composite operators. Because differences in effective regularization strength during training may also affect the results, we describe the comparison conditions and the preservation weight sweep in our response to Q3 below.

PAF’s performance should be evaluated in terms of both recognition accuracy and forgetting. In Order 2 of Table 2, forgetting was 0.55 and 0.90 percentage points for C-LoRA and PAF, respectively, and their Last WERs were 15.47% and 14.57%. In Order 4, the corresponding values were 1.53 and 2.79 percentage points for forgetting and 18.33% and 17.48% for Last WER. These results preclude a general claim that PAF achieves the lowest forgetting in every setting, and the lower final recognition error is consistent with our objective of improving retention and adaptation together.

W4. Scope of the evaluation and scalability to longer sequences

We trained on sequences of up to six English speech domains. During adaptation to the sixth domain, we retained all five earlier domains in the preservation objective, and PAF achieved a Last WER of 17.48%, compared with 18.33% for the best baseline. This shows that PAF’s performance advantage persisted through repeated adaptation at the sequence length we evaluated.

PAF retains PCA summaries and low-rank V/O update histories for each domain, so both storage and the cost of computing the preservation loss grow with the number of domains. Our results therefore apply to the English ASR sequences evaluated here, and performance on substantially longer sequences and under a fixed memory budget remains to be evaluated. In the paragraph on limitations and future work in Section 6 of the revised manuscript, we clarified this scope and identified evaluation on longer sequences and management of summaries and update histories under a fixed memory budget as future work.

Q3. Preservation strength, computational budget, and Pareto analysis

In Whisper-small Order 3, PAF and Individual trained the same Q/K/V/O LoRA parameters using the same training data, batch size, learning rate schedule, number of updates, and Q/K/V PCA support dimension (m=16m=16). Individual used a separate O-input support to evaluate the O projection independently. We evaluated λ∈{0.1,0.5,1.0}\lambda\in\{0.1,0.5,1.0\} for both methods. Because PAF uses a radius weighted composite penalty and Individual uses unweighted projection level penalties, the two methods can differ in effective regularization strength at the same λ\lambda. We matched the number of trainable parameters and optimizer updates, and a comparison that directly matches the regularization gradient norm and GPU time remains open.

To examine retention and adaptation together, Table 12 in Appendix A.4 reports mean WER on the two previous domains and WER on the new domain, TED2, both evaluated at the same final checkpoint for each setting.

Methodλ\lambdaOld domain WER (↓)New domain WER (↓)
Individual0.122.799.56
Individual0.524.049.83
PAF0.120.849.16
PAF0.521.449.34

Across the four settings evaluated, both PAF settings achieved lower old domain and new domain WER than both Individual settings. PAF at λ=0.1\lambda=0.1 was the only Pareto non dominated setting among the four, with old domain WER 1.95 percentage points lower and new domain WER 0.40 points lower than the best Individual setting. In Table 11, which also includes λ=1.0\lambda=1.0, every PAF setting had lower forgetting than every Individual setting. These results show a favorable retention–adaptation trade-off for PAF in this single seed Order 3 sweep. In Appendix A.4 of the revised manuscript, we specified the matched training conditions and the differences in the two preservation penalties before Table 11 and reported the Pareto comparison among the four evaluated settings immediately after Table 12.

Q4. Cost as the number of domains grows and adaptation under a fixed memory budget

At a fixed PCA dimension and LoRA rank, PAF’s retained storage grows linearly with the number of domains. The size of each domain’s summary is independent of its recording duration, and each new domain adds a summary and a low-rank V/O update history. Each previous domain also contributes a term to the preservation loss during training, increasing its computational cost.

Table 15 quantifies the trade-off between recognition performance and storage when we reduce the support dimension. In Whisper-small Order 3, reducing mm from 16 to 8 reduced retained storage by 39.1%, from 15.48 MiB to 9.43 MiB, and increased Last WER by 0.90 percentage points, from 16.63% to 17.53%. This measures the effect of allocating less storage to each domain’s summary.

Adding domains under a fixed total memory budget requires managing the growing V/O update history alongside any reallocation of support dimensions or compression of existing PCA summaries. Discarding a domain’s summary removes its direct preservation term, so the effect on retention also needs to be evaluated. Whether PAF retains its advantage under online compression and summary removal remains open. Appendix B.4 describes what the support dimension sweep measures, and Section 6 identifies fixed budget management of summaries and update histories as future work.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revision

Acknowledgements

We sincerely thank the reviewers for their careful reading and constructive feedback. Their comments helped us explain more clearly what PAF preserves and strengthen our evaluation of its effect on recognition performance.

Major Revisions

  1. Clarified the research question and its relation to prior work. Section 1 now frames the central question of how to measure changes in context selection (QK) and information transport (VO) on earlier speech domains without retaining their utterances. We explain how PAF addresses this question in relation to GainLoRA, SEEKR, and Attention-Spectrum. Section 4.1 also explains the rationale for the five main domain orders.

  2. Clarified the scope of the theoretical guarantees. Section 3.3 and Appendix C.2 explain how the bound on attention-output changes motivates the PAF loss and how support residuals enter the bounds on logit changes and increases in mean token negative log-likelihood (NLL). We also state the conditions sufficient to preserve the decoded transcription under deterministic decoding. In our response to Reviewer NijA, we use Figure 3 to discuss the relationship between operator drift and forgetting on LibriSpeech. We also report support residuals, representation drift, observed logit changes and their upper bounds, and WER changes on the same development utterances.

  3. Expanded the analysis of adaptation along shared input directions and the comparison with Individual. Table 9 in Appendix A.3 shows how the preservation weight affects Q/K/V updates along shared directions and NLL on previous and new domains. Appendix A.4 specifies the trainable parameters and number of optimizer updates shared by PAF and Individual, explains how the two methods weight their preservation penalties, and adds a Pareto comparison of WER on previous and new domains (Table 12).

  4. Added baseline comparisons and a paired significance test against the best baseline. We added comparisons with two replay-free continual ASR methods, CSSVD and Ma-RPGP (Appendix D, Table 20), and with unadapted Whisper-small (Appendix B.4). Section 4.3 and Appendix D describe our ASR implementation of Attention-Spectrum. For Whisper-small in Order 3, we performed a paired utterance bootstrap test to compare Avg WER between PAF and O-LoRA, the best-performing baseline on this metric (Section 5.1 and Appendix A.5).

  5. Clarified PCA support coverage, hyperparameter selection, and storage costs. Appendices B.2–B.4 distinguish the fraction of projected inputs that exceed the stored radii from the fraction of input variation captured by the PCA subspaces. We explain how we selected m=16m=16 and λ=1\lambda=1 using development data and kept both fixed throughout the main experiments. Table 14 reports an additional development-set evaluation of the preservation weight, and Table 15 shows how recognition performance, forgetting, and storage vary with the support dimension.

  6. Evaluated VO reconstruction on an additional six-domain sequence. In Appendix C.3, we measured the error in reconstructing VO reference operators after repeated merges on a randomly ordered six-domain sequence (Table 17). We evaluated FP64, FP16, and BF16 reconstruction. The maximum relative error was below 1% in both FP16 and BF16. Table 18 reports recognition performance and forgetting for the four methods evaluated on the same sequence.

  7. Expanded the discussion of scalability and future work. Following Reviewer G7xT’s suggestion, we moved the limitations from Appendix E to Section 6, where we explain how retained storage and the cost of computing the preservation loss grow with the number of domains. We identify the management of PCA summaries and V/O update histories under a fixed memory budget, evaluation on longer sequences and multilingual speech, and extensions to CTC and transducer models as directions for future work.

We hope that the revised manuscript and our point-by-point responses have addressed the reviewers’ questions and concerns. If we have left any point unclear or misunderstood any of the reviewers’ comments, we would be grateful if the reviewers could let us know so that we can address it more fully during the discussion. We sincerely appreciate the time and care the reviewers have devoted to our work, and we are grateful for the thoughtful comments that guided these revisions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.