- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies replay-free continual automatic speech recognition, in which one ASR model is adapted to a sequence of acoustic domains using only current-domain data, with no stored past utterances, no task identifier, and a single merged model at inference. Sequential LoRA adaptation is parameter efficient, but each new update overwrites earlier domains, and existing continual-LoRA methods such as O-LoRA, InfLoRA, C-LoRA, and GainLoRA constrain the parameter-update space through orthogonality, subspace isolation, or gating. The authors instead propose Preserving Attention Functions (PAF), which preserves the function that attention performs rather than the parameters themselves. Each attention head is decomposed into a QK operator for context selection and a VO operator for information transport. After every domain, PAF builds a compact summary of the head's query and key/value inputs using PCA bases, means, and radii, which provides fixed coordinates that are independent of utterance length. During later adaptation it regularizes the drift of both composite operators on these supports, with loss weights derived from a bound on the attention-output change. Score references are stored directly, while transport references are reconstructed from the merged weights and a history of low-rank V/O updates, so the method needs neither replay nor a separate teacher model. All updates merge into a single model with about 15.48 MiB of additional storage and unchanged inference speed. Experiments on Whisper-small, OWSM, and S2T-Large across several domain orders and against six baselines show lower final and average WER and reduced forgetting, supported by component ablations and diagnostics on PCA versus random supports, shared cross-domain input directions, and storage and runtime cost.
Critical strengths
1.The paper makes a well-motivated move from constraining the LoRA update space to preserving the attention function itself. It regularizes the QK and VO composite operators on fixed, utterance-length-independent PCA supports rather than freezing the four projections separately, which permits coordinated internal changes while keeping the net function on old domains. This gives replay-free and teacher-free retention that merges into a single model with about 15.48 MiB overhead and unchanged inference speed (RTF 0.389).
2.The validation is thorough. It spans three ASR backbones, several domain orders, three seeds, component and efficiency analyses, and the unfavorable S2T result, where average WER is worse than GainLoRA, is reported rather than concealed.
Critical weaknesses
1.The theoretical guarantee does not hold under the paper's own experimental setting, yet the framing suggests a provable preservation result. Propositions 1 and 2 require zero preservation error and exact coverage of every past input within the bounded supports. With m = 16 the supports explain only about 46% of the energy, some inputs lie outside the radii, and the layer-wise constants in the full-model bound are never quantified. The method is therefore in practice a theory-motivated empirical regularizer, and the bound should not be read as a WER guarantee.
2.The advantage over the strongest baselines is not shown to be robust. On several orders the gain over the best baseline is only 0.6 to 0.85 WER points, and the main table reports standard deviations without a paired significance test. The closest replay-free attention baseline, Attention-Spectrum, was adapted to ASR by the authors rather than tuned by its original authors, which weakens the claimed superiority over attention-level retention, while the core "shared directions do not impede new learning" argument rests mainly on correlational evidence and the 45% NLL result itself reveals an unresolved tension on those directions.
Questions for authors
1.Propositions 1–2 assume zero operator error and exact coverage of every past input within the support radii, yet at m = 16 only about 46% of the energy is explained and some inputs lie outside the radii. Please report (i) the fraction of test inputs/positions that violate the radius bounds, (ii) the numerical value of the Prop. 1–2 bound on the real 36-block network and how the layer-wise constants accumulate, and (iii) whether any probabilistic or expected guarantee survives when coverage is incomplete. If not, please state explicitly that the theory motivates rather than guarantees preservation.
2.On several orders the margin over the best baseline is only 0.6–0.85 WER points. Can you provide a paired significance test (paired utterance bootstrap or a paired test across seeds) for PAF versus the best baseline in each order, with confidence intervals, so the margins can be judged against seed and utterance noise?
3.Removing Q/K/V updates along shared directions reduces the new-domain NLL gain by 45%, which also indicates that these directions matter for learning the new domain. How much of each current update lies inside versus outside the retained supports, and how does the new-domain NLL gain change as λ increases?
4.How were m = 16 and λ = 1.0 chosen, and are they fixed across all orders and backbones without access to later-domain test data?
5.Equation (46) recovers past V/O by subtracting accumulated low-rank updates. Appendix A.1 mentions FP64 over 36 blocks. How large is the error between the reconstructed T_s and the T_s computed directly from the original checkpoint, in FP64 and in the FP16/BF16 precision a deployment would use, and does this error accumulate over six domains?
4 comments
W2. Robustness of the performance gains, comparison with Attention-Spectrum, and evidence for adaptation in shared directions
We interpret PAF’s gains within the scope of the backbones, domain orders, and metrics evaluated. Table 2 reports means and standard deviations over three seeds, and PAF had the lowest mean Last WER and Avg WER in all four Whisper-small orders. We also used a paired utterance bootstrap in Order 3 to evaluate the difference in Avg WER between PAF and O-LoRA. Q2 below gives the procedure and scope of this test.
We evaluated our ASR implementation of Attention-Spectrum. Section 4.3 specifies the regularization weight , and Table 19 in Appendix D describes the retained spectral and angular statistics and the preservation penalty. The comparison therefore applies to this ASR implementation and evaluation setting. We clarified the ASR adaptation and its evaluation setting in Section 4.3 and Appendix D so readers can assess the scope of the comparison.
The intervention in Table 8 shows that updates along shared input directions contributed to learning the new domain. Removing the Q/K/V update component along past input directions and matching the total update norm reduced the one step improvement in TED2 NLL by 45.13%. To examine these directions during adaptation, we compared from the same checkpoint. Increasing from 0 to 1 reduced the share of update energy in the retained subspace from 15.36% to 4.12%, while the decrease in TED2 NLL grew from 1.0478 to 1.2254, and the mean NLL increase on the two past domains fell from 0.3145 to 0.1072. On this transition, PAF retained updates along shared directions while improving both retention and adaptation. Establishing the causal contribution of individual compensating Q/K components requires further analysis, and in Appendix A.3 we limited our interpretation of shared directions to the transition evaluated in Table 9.
Q2. Paired significance test against the best baseline
We used a paired utterance bootstrap to compare Avg WER between PAF and O-LoRA on Whisper-small in Order 3. O-LoRA had the lowest three seed mean Avg WER among the baselines in this order (Table 2). With the seed 2022 checkpoints fixed, PAF’s Avg WER was 0.645 percentage points lower than O-LoRA’s, with a 95% confidence interval for the improvement (O-LoRA minus PAF) of [0.448, 0.845] percentage points.
Within each domain, we sampled evaluation utterances with replacement, keeping both methods’ decoding outputs from all three adaptation stages together. This preserved the pairing across methods and stages. In each bootstrap replicate, we recomputed corpus WER for all nine combinations of domain and stage and used Equation (24) to calculate the difference in Avg WER. The entire confidence interval lies above zero, indicating significantly lower Avg WER for PAF when accounting for variation across evaluation utterances with the two checkpoints fixed.
The 0.6–0.85 percentage points mentioned by the reviewer refer to improvements in Last WER across several orders. Our additional test covers Avg WER in Order 3 for the seed 2022 checkpoints, and tests covering Last WER differences in each order and variability across training seeds remain open. Section 5.1 now mentions this test, and Appendix A.5 reports the bootstrap procedure, confidence interval, and scope of the test.
Q3. Share of update energy inside and outside the retained support and the effect of the preservation weight
Updates along past input directions contributed to adaptation on TED2. Removing the Q/K/V update component along those directions and rescaling the remaining update to the original total norm reduced the one-step improvement in TED2 NLL by 45.13% (Appendix A.3, Table 8). PAF allows learning along these directions by penalizing changes in the QK and VO operators on past inputs.
To measure the share of updates along shared directions during training, we adapted to TED2 from the same checkpoint after LS and CHiME3, varying only . Following Equations (36)–(37), we decomposed the AdamW updates to the Q/K/V weights into a component in the span of the PCA bases and input means of the two past domains and a component in its orthogonal complement. We computed the energy shares below by summing the squared Frobenius norms of each component over 142 updates sampled at intervals of 100 steps. We measured the decrease in TED2 NLL from the same starting value of 1.5917 in all three settings.
| Energy in past subspace | Energy in orthogonal complement | Final TED2 NLL (↓) | Decrease in TED2 NLL (↑) | Mean NLL increase on the two past domains (↓) | |
|---|---|---|---|---|---|
| 0 | 15.36% | 84.64% | 0.5439 | 1.0478 | +0.3145 |
| 0.1 | 5.56% | 94.44% | 0.4493 | 1.1424 | +0.1107 |
| 1 | 4.12% | 95.88% | 0.3663 | 1.2254 | +0.1072 |
On this transition, increasing from 0 to 1 reduced the share of update energy along shared directions from 15.36% to 4.12% and yielded a larger reduction in TED2 NLL, from 1.0478 to 1.2254. The mean NLL increase on the two past domains also fell from 0.3145 to 0.1072. At the default , some update energy remained along shared directions, and the new domain showed the largest NLL decrease among the three settings. We added the measurement procedure and results to Appendix A.3 (Table 9).
Q4. Choice of PCA dimension and preservation weight and use of test data
| Mean WER on the two past domains | WER on the new domain (TED2) | Mean WER on all three domains | |
|---|---|---|---|
| 0.1 | 11.03 | 10.60 | 10.89 |
| 0.5 | 11.26 | 10.55 | 11.02 |
| 1.0 | 10.80 | 10.28 | 10.63 |
As described in Section 4.1 of the submitted paper, we split the adaptation data into training and development subsets in a 9:1 ratio and used the development subset for validation. The training, development, and test subsets were disjoint, and we used the same splits for all methods and seeds. We fixed and across all main experiments, without retuning either value for individual domain orders or backbones.
We selected and using development results and fixed them before the main evaluation, without using test results from later domains to select or retune either hyperparameter.
Table 15 reports the performance–storage trade-off for , and Table 13 reports sensitivity to the preservation weight. In an additional evaluation of the existing checkpoints on 1,015 development utterances, again gave the lowest mean WER on the past domains and the lowest WER on the new domain. This evaluation provides further support for the default setting.
Q5. VO reference reconstruction error across precisions and error accumulation
The submitted paper evaluated one six domain training order. To test whether the VO references could be accurately reconstructed after repeated merges in another order, we ran an additional experiment on Whisper-small with a random ordering of the six domains. The training seed was 2024, and the order was Common Voice → GigaSpeech-YOU → CHiME3 → GigaSpeech-POD → TED2 → LibriSpeech-other.
-
Reconstruction error by precision after repeated merges. We reconstructed the past VO references using Equations (46)–(47) from the checkpoint after merging the final domain’s updates. We compared each reconstructed reference with the FP64 reference computed directly from the original checkpoint for that domain. The table reports the maxima over 180 comparisons across five past domains and 36 attention blocks. We computed the relative error as .
Reconstruction precision Maximum relative Frobenius error Maximum absolute elementwise error FP64 FP16 BF16 Reconstructing the reference for Common Voice, the first domain, requires subtracting the V/O updates of all five subsequent domains from the final merged weights. Including this longest update history, the maximum relative error was 0.084% in FP16 and 0.700% in BF16. The final relative reconstruction error thus remained below 1% after five subsequent merges.
-
Recognition performance on the same domain order. We evaluated all methods on the same 11,678 utterances.
Method Last WER (↓) Avg WER (↓) Forgetting (↓) IncLoRA 25.23 21.93 10.06 O-LoRA 20.81 19.61 6.56 GainLoRA 19.68 19.16 5.36 PAF 19.49 18.53 5.13 On this additional six domain order, PAF had the lowest Last WER, Avg WER across all adaptation stages, and mean forgetting among the four methods. Compared with GainLoRA, the strongest baseline, PAF reduced Last WER by 0.19 percentage points, Avg WER by 0.63 points, and mean forgetting by 0.23 points.
Thank you for your detailed feedback on our work.
W1. Conditions for the theoretical guarantees and scope of the empirical results
As explained in Section 3.3, the conditional bound on changes in the attention function motivates the design of the PAF loss as a surrogate objective and the choice of its coefficients. The gains in recognition accuracy and reductions in forgetting are empirical, and we assessed the practical value of the objective through component comparisons and a controlled intervention.
Proposition 1 bounds the squared norm of the change in a head’s attention output by its preservation loss when the current and reference heads receive the same inputs on the bounded stored support and use the same allowed keys and additive mask.
∥fθ,s,h−fs,h∥22≤ℓs,h(θ).
This inequality also holds for positive preservation loss. Proposition 2 extends the analysis to the complete model when the current and reference models differ only in the regularized attention projections and use the same acoustic input, decoder prefix, and attention masks. Under finite sensitivity bounds, Equations (42)–(44) bound the change in logits and the increase in mean token NLL, and Equation (43) accounts for incomplete coverage through the support residual ρs,ℓ(a,p).
Zero preservation error and complete coverage are sufficient conditions for exact recognition preservation, as stated after the proof of Proposition 2. Under these conditions, the same deterministic decoding procedure and tie breaking rule preserve the transcription and its WER. We therefore distinguish the conditional bounds in Propositions 1 and 2 from the sufficient conditions for exact recognition preservation.
The experiments also show the effect of PAF under limited input coverage. At the same m=16, PAF reduced forgetting relative to T only from 1.55 to 0.90 percentage points in Order 2 and from 6.32 to 3.67 percentage points in Order 3. PAF also had lower Last WER, Avg WER, and forgetting than Individual in both orders (Table 4). We clarified the relationship between the theoretical conditions and the observed performance gains in Section 3.3 and Appendix C.2.
Q1. Radius violations, the numerical bound, and the guarantee under incomplete coverage